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Overview 


Early math has been shown to predict not only longer-term math achievement, but also future 
reading achievement, high school completion, and college attendance. Yet effects from early math 
programs often fade out as children move into more varied instructional contexts in elementary 
school. This fade-out suggests the need for an alignment of math instruction across the early years to 
support children’s earlier math gains. 


In 2011, the Robin Hood Foundation began a collaboration with MDRC to rigorously test an 
approach aimed at boosting early math skills. High 5s, a supplemental small-group math club 
program for children in kindergarten, was designed specifically as a follow-on to the Making Pre-K 
Count study, in which preschools implemented an enriched, evidence-based math curriculum 
(Building Blocks) supported by professional development for teachers. High 5s aligned with 
Building Blocks by focusing on children’s developmental progression, encouraging hands-on 
learning, supporting student reflection about mathematical thinking, and using formative assessment 
(which helps teachers modify their approaches) and instruction differentiated by children’s ability 
levels. As described in more detail in a companion report, children who were offered two years of 
enhanced math instruction (High 5s and Making Pre-K Count) had stronger math skills than those 
who had no enhanced math in pre-K or kindergarten — an impact equivalent to more than four 
months of growth — as well as more positive attitudes toward math. Given these encouraging 
findings, this report describes what was needed to implement the High 5s program successfully. 


Key Findings 


When the project team set out to design a math enrichment program, there were uncertainties about its 
feasibility: Would schools be receptive? Could an appropriate time and place to hold the clubs be 
identified? Would children attend regularly and be engaged? Could enough facilitators be hired at a 
paraprofessional level salary and retained for an entire year? With the support of a strong training and 
supervision model, all these issues were addressed and the High 5s program met all the benchmarks 
identified at the outset. Throughout the year, attendance and engagement were high, sessions were held 
regularly, and activities were implemented as intended by a team of committed facilitators. 


All students who were part of the High 5s study also received typical math instruction in their 
kindergarten classrooms. Classroom instruction was found to differ in a number of ways from the 
instruction in High 5s: Students participating in High 5s were presented with a wider range and 
somewhat more advanced instructional content than was observed in classrooms, that content was 
delivered in a small-group format as opposed to the whole-class format in which most classroom 
mathematics was delivered, and facilitators engaged in more open-ended questions and more 
differentiation of material than was observed in classrooms. 


These findings suggest that small-group enrichment may be one way to provide kindergarten 
instruction that is more closely aligned with the pre-K experience, which typically involves a 
substantial amount of small-group instruction with many hands-on learning opportunities. The math 
club model may help ensure that children get more individualized instruction, and its alignment with 
the preschool experience may be one way to help mitigate fade-out of math gains after preschool. 


ill 


Contents 


Overview 

List of Exhibits 
Preface 
Acknowledgments 
Executive Summary 


Chapter 


1 Introduction 
Fade-Out of Early Childhood Impacts 
The High 5s Math Club Program 


2 The High 5s Program 
The Structure of the High 5s Program 
Communication with Families and Teachers 
The Learning Trajectory Framework 
Formative Assessments 
Summary 


3 Staffing the High 5s Program 
Training and Supervision 
Supervision Model 
Summary 


4 Quality and Fidelity of Implementation 
Implementation Data 
Implementation Findings 
Summary 


5 Instruction in the Kindergarten Classrooms Compared with High 5s 
The Kindergarten Classrooms 
Math Instruction in the Classrooms Compared with High 5s Clubs 
Summary 


6 Implications of the Findings 
Potential Contributions of the High 5s Program 
Program Replicability 
Conclusions 


ili 
vii 
1x 

Xi 
ES-1 


NNR 


Appendix 
A High 5s Sample Lesson 
B_ Curriculum Pacing Guide and Sample Activities 


C_ Club Session Log Content 


References 


Other MDRC Publications on Making Pre-K Count and High 5s 


vi 


49 


57 


71 


83 


85 


Table 


3.1 
4.1 


4.2 


4.3 
5.1 
ay 


5.3 


5.4 


5.5 


6.1 


Figure 


2.1 
22 
4.1 
4.2 
5.1 


List of Exhibits 


Facilitator Demographics 


Amount of Time Spent on Math in High 5s Clubs, Based on Facilitator Logs 
and Spring Club Observations 


Fidelity to the Model: Instructional Quality, Based on Facilitator Logs and 
Club Observations 


Student Engagement, Based on Facilitator Logs and Spring Club Observations 
Description of the Kindergarten Classroom, Based on Classroom Observations 


Quality of Math Instruction in Kindergarten, Based on Classroom 
Observations 


Activity Type and Materials Used During Math Instructional Time, 
Based on Classroom Observations 


Nature of Math Instruction in Kindergarten Classrooms and High 5s Clubs, 
Based on Spring 2016 Observations 


Percentage Breakdown of Content During Math Instructional Time in 
Kindergarten Classrooms and High 5s Clubs 


Potential Resources Needed to Replicate the High 5s Program During the 
School Day Using School-Based Staff 


Sample Lesson 

High 5s Counting Trajectory 

High 5s Club Attendance Rates 

Facilitators Meeting and Exceeding Expectations of Implementation Quality 


Instructional Content in Kindergarten Classrooms in Fall 2015 


vii 


12 


21 


22 
26 
31 


33 


35 


36 


37 


44 


Preface 


In 2011, MDRC entered into a collaboration with the Robin Hood Foundation to identify early 
childhood interventions that had the potential to change the life trajectories of New York City 
children experiencing poverty. The effort began with a focus on improving early math skills, an 
area shown to be among the most important predictors of a young child’s later academic 
achievement. In a rigorous test, prekindergarten classrooms in New York City were provided 
with a high-quality, evidence-based math curriculum (Building Blocks) and ongoing teacher 
training and coaching over two years. The program was named Making Pre-K Count. 


Though few studies of pre-K math curricula follow children over the long term, those 
that do often show a fade-out of math impacts as children move into elementary school. Various 
theories exist as to why fade-out occurs. If shifting educational contexts as children progress 
from preschool through high school are a factor, aligning children’s instructional experiences 
across school years may help maintain pre-K gains. With this in mind, the study team set out to 
explore whether a kindergarten enrichment program that was aligned with the pre-K experience 
in terms of both content and pedagogy could provide a critical boost that would lead to more 
sustained long-term achievement gains. That initiative, a small-group math club program for 
kindergartners called High 5s, which was developed and piloted in 2014 and implemented in 
2015, is the focus of this report. 


The project faced a number of uncertainties: Would schools be receptive? Would it be 
possible to successfully recruit a team of facilitators to implement the program? Would schedul- 
ing and space issues stymie the effort? And if those issues were resolved, would kindergartners 
come regularly and be engaged in a supplemental math club? But the High 5s program met its 
implementation goals. Throughout the year, attendance and engagement were both high, 
sessions were held on a regular basis, and a team of committed facilitators delivered the 
activities as designed. The results show promise: As described in more detail in a companion 
report, students who participated in both Making Pre-K Count in pre-K and High 5s in kinder- 
garten had stronger math skills at the end of kindergarten than students who had received no 
enhanced math instruction, an effect equivalent to more than four months of math learning. 


This report provides detailed information about the implementation of the High 5s pro- 
gram: what went well, what barriers existed, and the factors that may have contributed to its 
success. 


Gordon L. Berlin 
President, MDRC 
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Executive Summary 


Early math is a strong predictor not only of longer-term math achievement, but also of later 
reading achievement and even high school completion and college attendance.! Thus, boosting 
children’s early math skills is thought to be a way to improve long-term outcomes across a wide 
range of domains. Working from this notion, MDRC and the Robin Hood Foundation, one of 
New York City’s leading antipoverty organizations, began a joint effort to rigorously test 
promising early childhood interventions that might make a lasting difference for children 
growing up in poverty. The early part of the initiative was focused on trying to boost that 
potential “linchpin” outcome — early math skills. 


A two-pronged approach to boosting early math skills was planned. The first part, 
called Making Pre-K Count, provided prekindergarten classrooms in New York City with a 
high-quality math curriculum (Building Blocks) and ongoing teacher training and coaching over 
two years. Building Blocks is a 30-week, evidence-based curriculum designed to take into 
account and build on children’s natural developmental progression in math skills; it also 
encourages instructional practices such as hands-on learning, student reflection about mathe- 
matical thinking, formative assessment to measure student progress and inform teaching 
decisions, instruction differentiated by the skill level of individual children, and a mix of small- 
and whole-group activities. 


The second part, a math club program called High 5s, is the focus of this report. The 
High 5s math clubs were designed specifically as a follow-on to the larger Making Pre-K Count 
study and were developed using the learning trajectories research on which the Building Blocks 
curriculum is based. The program was intended to provide a small-group math enrichment 
experience that was aligned with both the content and approach of the Building Blocks curricu- 
lum, for kindergarten students who had experienced Making Pre-K Count in pre-K. 


The High 5s program was developed by staff members at the University of Michigan 
with input from Doug Clements and Julie Sarama, the developers of the Building Blocks 
curriculum, and was implemented by Bank Street College of Education, which hired, trained, 
and supervised the club facilitators. The implementation included a number of key components: 


‘Greg J. Duncan, Chantelle J. Dowsett, Amy Claessens, Katherine Magnuson, Aletha C. Huston, Pamela 
Klebanov, Linda S. Pagani, Leon Feinstein, Mimi Engel, and Jeanne Brooks-Gunn, “School Readiness and 
Later Achievement,” Developmental Psychology 43, no. 6 (2007): 1428-1446; Greg J. Duncan and Katherine 
Magnuson, “The Nature and Impact of Early Skills, Attention, and Behavior” (paper presentation, Russell Sage 
Foundation Conference on Social Inequality and Educational Outcomes, New York City, 2009). 


e A total of 24 paraprofessional-level facilitators (mostly recent college gradu- 
ates with limited formal teaching experience) administered the program to 
over 300 children in 24 New York City public schools during the 2015-2016 
school year. The clubs typically included four children and one facilitator. 


e Facilitators received substantial training on the curriculum and instructional 
approach before the start of the program as well as weekly coaching and su- 
pervision from the Bank Street supervisors and program director that was 
designed to be responsive to their individual needs. 


e Clubs took place during noninstructional time (before school, after school, or 
during lunch). They were designed to meet three times a week for approxi- 
mately 28 weeks starting in the fall of 2015, with each club session lasting 
approximately 30 minutes. 


e Activities in the clubs were designed to move children along key mathemati- 
cal learning trajectories and were delivered in a game-like format that was 
intended to be fun, engaging, and developmentally appropriate. 


As described in more detail in a companion report, students who had two years of en- 
hanced math instruction (both Making Pre-K Count in pre-K and High 5s in kindergarten) had 
stronger math skills (on the REMA-K,?” one of two measures of math achievement in the study) 
at the end of kindergarten than students who had received no enhanced math instruction in 
either pre-K or kindergarten, an effect equivalent to 4.2 months of additional growth in math 
skills. Students in High 5s also had stronger math skills as measured on the REMA-K than 
those who had received only one year of enhanced math instruction as part of the Making Pre-K 
Count project, an impact equivalent to approximately 2.5 months of growth. 


With these positive findings as a backdrop, this report describes the High 5s program in 
detail, noting what was needed to implement the program with fidelity to the model. The 
program set out to achieve a number of goals — it was designed to build on the content of the 
Building Blocks curriculum; to deliver hands-on, engaging instruction in a game-like format; 
and to provide opportunities for both differentiated instruction and for children to delve deeply 
into mathematical concepts by explaining their mathematical thinking to others. Because the 
High 5s program was new and developed specifically for this project, and because it aimed to 


°The Research-Based Early Math Assessment-Kindergarten (REMA-K) is a version of the full REMA. 
For more information on the impact measurement, see Shira K. Mattera, Robin Jacob, and Pamela A. Morris, 
Strengthening Children’s Math Skills with Enhanced Instruction: The Impacts of Making Pre-K Count and 
High 5s on Kindergarten Outcomes (New York: MDRC, 2018). 


3Mattera, Jacob, and Morris, Strengthening Children’s Math Skills. 


achieve an ambitious set of logistical and educational objectives, it was not clear at the outset 
whether High 5s could be implemented successfully. 


Despite these uncertainties, the High 5s program met all the benchmarks identified for 
the program at the outset of the study, and with the support of a strong, ongoing training and 
supervision model, it was implemented with fidelity. Findings from an analysis of implementa- 
tion data indicate the following: 


e Student attendance was high, with an average attendance rate of 87 per- 
cent across the year. The majority of students attended most of their club ses- 
sions, with more than 90 percent of the students having attendance rates over 
70 percent. Over 80 percent of students had attendance rates over 80 percent. 


e Club sessions were completed as scheduled and the curriculum was de- 
livered as intended. Ninety-two percent of scheduled sessions were com- 
pleted and club observations indicated that 96 percent of the activities were 
implemented as intended. 


e Students spent an average of 25 minutes in each club on mathematics. 
The goal was to spend at least 20 minutes of the math club on math. Students 
spent about 75 minutes a week on math in addition to the time they spent on 
math in the kindergarten classroom. 


e Facilitators created a positive instructional climate in clubs. All facilita- 
tors met expectations for having a good rapport with students. Over 80 per- 
cent met expectations for making math learning fun and using positive strat- 
egies to manage behavior. This was true even though a number of facilitators 
cited managing student behavior as one of the most challenging aspects of 
the job. 


e Student engagement was also high (as reported by both facilitators and 
observers). Engagement was rated on a scale of | (all students are disen- 
gaged) to 5 (all students are engaged). Average facilitator-rated engagement 
scores were high, with nearly all children engaged in most clubs: Ratings 
ranged from 4.7 to 4.9 for the different types of High 5s activities. 


e While the program was generally implemented with fidelity to the model, 
ensuring the highest quality of instruction — instruction that included 
differentiation, involved reflective questioning, and underscored the 
mathematical objective of the activities — was more challenging, as it 
was for the kindergarten teachers in our sample, despite ongoing and high- 
quality training and coaching. 


All students who were part of the High 5s study also received typical math instruction 
in their kindergarten classrooms. To better understand the potential value that the High 5s 
program added, kindergarten classrooms were observed to understand the nature of the mathe- 
matics instruction students were receiving and how it compared with the instruction students 
were receiving in High 5s clubs. Instruction in kindergarten classrooms was found to differ in a 


Overall, the logistics of the program proved to be the most challenging 
aspect of running the clubs. The most common logistical challenges in- 
cluded finding an appropriate time and place to meet, escorting children to 
and from clubs, and having facilitators who traveled from school to school. 


number of ways from the instruction in High 5s: 


Overall, despite uncertainty in the planning phases about the viability of a math club 
model like High 5s, the program was implemented well, with high attendance rates and strong 
student engagement. Staff members with a variety of backgrounds and experiences but limited 
formal teaching experience were able to deliver a wide range of math content with adequate 
instructional quality. The program had a robust, statistically significant impact on one of two 


The High 5s program presented a wider range and somewhat more ad- 
vanced instructional content than was observed in kindergarten class- 
rooms. Classroom teachers were, for the most part, following the district’s 
Common Core-aligned kindergarten math curriculum, which focused on 
numbers and operations for a majority of the year. Geometry, measurement, 
and patterning activities were included throughout the year in High 5s. 


Content in the High 5s clubs was delivered in a small-group, hands-on 
format, while small-group instruction in the kindergarten classroom 
tended to be limited. By design, the High 5s clubs were small groups, and 
all the High 5s activities involved the use of hands-on manipulatives. In the 
kindergarten classrooms, small-group activities accounted for around 5 per- 
cent of the activities observed, and about two-thirds of the activities observed 
in kindergarten classrooms involved only workbooks or no materials at all. 


Observations indicated that High 5s facilitators asked more open-ended 
questions and engaged in more differentiation of material than was ob- 
served in kindergarten classrooms. 


measures of math achievement. 


Findings from this study suggest that small-group enrichment may be one 
way to provide kindergarten instruction that is more closely aligned with the 


pre-K experience, which typically involves a substantial amount of small- 
group instruction with many hands-on learning opportunities. 


e The small-group experiences that a math club model offers can also help en- 
sure that children get more individualized instruction and do not “fall through 
the cracks.” 


e It is worth considering whether experiences that are more closely aligned 
with the preschool experience and that emphasize differentiated instruction 
may help mitigate the fade-out often associated with high-quality preschool 
experiences. 


The positive impacts of the High 5s program demonstrated in this study suggest that 
further research is needed to examine whether and how it works in different contexts. For 
example, is the program effective in scenarios in which children come into kindergarten with a 
variety of different pre-K math experiences? Would it be effective if schools used in-house staff 
members to deliver the clubs, or could it be done without the level of training and supervision 
that facilitators were provided in the model that was tested? At the same time, future research is 
needed to parse how the different components of the program work and for whom. 


Chapter 1 


Introduction 


In 2011, MDRC and the Robin Hood Foundation, one of New York City’s leading antipoverty 
organizations, entered into a collaboration to identify and rigorously test promising early 
childhood interventions that had the potential to change the life trajectories of New York City 
children growing up in poverty. The decision was made to focus the initiative on trying to boost 
early math skills, which have been shown to be a strong predictor of later achievement for 
young children.' Early math skills predict not only later math achievement, but also reading 
achievement, executive function (the skills underlying self-regulation), and high school comple- 
tion and college attendance.” Math concepts, such as making comparisons, can help expand and 
enrich vocabulary as children use language to express and justify mathematical thinking.? And 
early math is thought to strengthen children’s executive function skills by requiring them to 
manipulate numbers and shapes and explain their thinking.* Thus, boosting children’s early 
math skills is thought to be a means to improve long-term outcomes across a wide range of 
domains. 


A two-pronged approach to improving early math skills was planned. The first ap- 
proach, called Making Pre-K Count, provided prekindergarten classrooms in New York City 
with a high-quality math curriculum (Building Blocks) and ongoing teacher training and 
coaching over two years. Building Blocks is a 30-week curriculum designed to take into 
account and build on children’s natural developmental progression in math skills.° The 
curriculum had demonstrated effectiveness in raising children’s early math achievement in a 
number of prior randomized trials.° For the study, 69 sites were randomly assigned either to 
implement two years of the curriculum and professional development or to continue their 
usual pre-K practices.’ 


‘Duncan et al. (2007); Duncan and Magnuson (2009). 

Duncan et al. (2007); Blair and Razza (2007); Duncan and Magnuson (2009). 
3Ginsburg, Lee, and Boyd (2008). 

4Clements, Sarama, and Germeroth (2016). 

5Clements and Sarama (2013). 

Clements and Sarama (2007, 2008); Hofer, Lipsey, Dong, and Farran (2013). 


7While pre-K sites implemented the program for two years, children in this study entered the classroom in 
the second year and received only one year of enriched pre-K math instruction. 


Fade-Out of Early Childhood Impacts 


Despite the apparent importance of early math, it is well known that many early childhood 
programs that show evidence of effectiveness at the end of the pre-K year see a fade-out of 
effects as children move out of the preschool setting and into later grades.* One hypothesized 
explanation for this fade-out is the lack of alignment between the instructional content and 
pedagogical approaches in pre-K and the instruction provided in elementary school.’ Research 
suggests that kindergarten teachers spend a significant amount of instructional time on basic 
number and shape concepts that students already know when entering kindergarten. !° Preschool 
programs are also often play-based and use a variety of manipulatives, and students frequently 
interact in small groups with a teacher or other students, while kindergarten classrooms may 
take a more didactic approach, with instruction occurring in a whole-group setting or with 
students working individually at their desks. To mitigate the potential fade-out of effects from 
early childhood programming that might result from such misalignment, and to test the benefit 
of providing an additional year of enhanced math instruction to children, a second initiative was 
designed. 


The High 5s Math Club Program 


The second initiative, a math club program called High 5s, is the focus of this report. The High 
5s program was designed specifically as a follow-on to the larger Making Pre-K Count study 
and was developed using the learning trajectories research on which the Building Blocks 
curriculum is based. The program was intended to provide a small-group math enrichment 
experience that was aligned with both the content and the approach of Building Blocks for 
kindergarten students who had experienced Making Pre-K Count in pre-K. The Building Blocks 
curriculum is designed to focus on children’s developmental progression or learning trajectories 
and also encourages such instructional practices as hands-on learning, student reflection about 
mathematical thinking, formative assessment to measure student progress and allow teachers to 
modify their approaches, instruction differentiated by children’s ability levels, and a mix of 
small- and whole-group activities.'!' High 5s was designed to include these key elements as 
well. Some of these practices are more easily implemented in pre-K classrooms than in kinder- 
garten classrooms because of smaller class sizes and smaller teacher-to-student ratios. 


Hofer, Lipsey, Dong, and Farran (2013); Puma et al. (2012). 
*Lee and Loeb (1995). 

‘OF ngel, Claessens, and Finch (2013). 

'\Clements and Sarama (2013). 


Before the math club model was selected, a number of different kindergarten interven- 
tions were considered. Small-group enrichment was chosen because there is substantial evi- 
dence regarding the effectiveness of small-group tutoring for young students in reading and of 
two-on-one tutoring with older students in math.’* Offering enrichment outside of regular 
classroom instruction also provided the study team with maximum flexibility to ensure that 
students’ pre-K and kindergarten experiences would be well aligned. Since the program was not 
being implemented in classrooms, the study team was able to develop curricular materials and 
train staff independently. The High 5s program paired about four children with one facilitator 
for each club, which met three times a week for approximately 30 minutes, either before or after 
school or during lunch. 


Making Pre-K Count and High 5s were developed as part of the Robin Hood Early 
Childhood Research Initiative, which was established to identify and rigorously test promising 
early childhood programs. The initiative is a partnership between Robin Hood and MDRC, a 
nonprofit, nonpartisan, education and social policy research organization. These studies, 
conducted in collaboration with Bank Street College of Education and RTI International, were 
also supported with lead funding from the Heising-Simons Foundation, the Overdeck Family 
Foundation, and the Richard W. Goldman Family Foundation. 


The High 5s study was designed to rigorously test whether the positive effects of a pre- 
K math program could be sustained if combined with math enrichment in kindergarten. As 
described in more detail in a companion report, students who were offered two years of en- 
hanced math instruction (both Making Pre-K Count in pre-K and High 5s in kindergarten) had 
stronger math skills (on the REMA-K," one of two measures of math achievement used in the 
study) at the end of kindergarten than those with no enhanced math instruction in either pre-K 
or kindergarten (effect size = 0.30 standard deviations).'* This is equivalent to 4.2 months of 
additional growth in math learning. Students who were offered two years of enhanced instruc- 
tion also had more positive attitudes toward math at the end of the kindergarten year than 
students who received no enhanced instruction.'> In a second comparison, two years of math 
enrichment (High 5s in kindergarten and Making Pre-K Count in pre-K) led to stronger math 
skills on the REMA-K (effect size = 0.19) compared with only one year of enhanced math 


?Slavin, Lake, Davis, and Madden (2010); Fryer (2011). 
>The Research-Based Early Math Assessment-Kindergarten (REMA-K) is a version of the full REMA. 
See Mattera, Jacob, and Morris (2018) for more information about the impact measurement. 


‘Effect size is calculated by dividing the estimated effect of the program (the difference between means 
for the program group and the control group) by the standard deviation for the control group. An effect size of 
0.30 therefore represents an improvement in math skills equal to 30 percent of the standard deviation. 


*Mattera, Jacob, and Morris (2018). 


instruction (Making Pre-K Count in pre-K but kindergarten as usual), an impact equivalent to 
approximately 2.5 months of growth. 


Given these positive findings, this report describes the High 5s program in detail, noting 
what was needed to implement the program with fidelity. Because the High 5s program was 
new and developed specifically for this project, and because it set out to achieve an ambitious 
set of logistical and educational objectives, it was not clear at the outset whether it could be 
implemented successfully. In the end, the program, supported by a strong, ongoing training and 
supervision model, was generally implemented with fidelity to the model. The program was 
also found to differ substantially from the math instruction already in place in kindergarten 
classrooms, suggesting that the High 5s program may have provided substantial added value to 
the children’s kindergarten math experiences. A number of important lessons about implemen- 
tation, attendance, and student engagement are described in this report. 


Chapter 2 describes the structural components of the program, and Chapter 3 details the 
staffing and associated training and supervision. The quality and fidelity of implementation is 
assessed in Chapter 4, and the math environment and instruction in the kindergarten classrooms 
are described and compared with High 5s in Chapter 5. The report concludes with a discussion 
about what might be needed to replicate the program in other contexts. 


Chapter 2 


The High 5s Program 


The High 5s program was developed by staff members at the University of Michigan with input 
from Doug Clements and Julie Sarama, the developers of the Building Blocks curriculum. The 
program had an ambitious set of objectives: It was designed to build on the content of the 
Building Blocks curriculum; to deliver hands-on, engaging instruction in a game-like format; 
and to provide opportunities for both differentiated instruction and for children to delve deeply 
into mathematical concepts by explaining their mathematical thinking to others. By delivering 
instruction in a small-group setting, the team hoped to avoid some of the challenges that make 
delivering differentiated instruction difficult in a large classroom with a limited number of 
adults. An initial version of the program was piloted in three schools during the 2014-2015 
school year, and revisions to the model were made based on lessons learned from the pilot. 


The Structure of the High 5s Program 


The High 5s clubs took place during noninstructional time (before school, after school, or 
during lunch). Each club consisted of about four children and was led by a trained facilitator 
who was hired and supervised by Bank Street College of Education. Bank Street also provided a 
high level of ongoing training and support for the facilitators throughout the year. Clubs were 
designed to meet three times a week for about 28 weeks starting in fall 2015, with each club 
session lasting approximately 30 minutes. Activities in the clubs were delivered in a game-like 
format and were intended to be fun, engaging, and developmentally appropriate. 


Each club session included two start-up activities and a main activity. The start-ups 
were meant to help students adjust to the club setting and reinforce key skills. Start-ups were 
usually repeated each session for one week, and together the start-ups were intended to last 
between 7 and 10 minutes. The main activities, designed to last 15 to 20 minutes, were the key 
component of clubs. Each main activity was intended to be repeated up to three times over the 
course of the year, but not in consecutive sessions. 


Each activity came with a mathematical objective, a materials list, a brief description of 
the activity, the mathematical development levels being targeted, and a picture of the set-up 
(Figure 2.1). This was followed by a semiscripted activity plan or instructions for playing the 
game. The instructions gave facilitators enough support to implement the activities with fidelity 
but also the flexibility to include adaptations. Suggestions were given for how to “scaffold” 
students at different levels of development (that is, how to provide assistance based on a 


Figure 2.1 


Sample Lesson 


Chocolate Chip Count 


Early Addition and Subtraction 


Students will subitize (know amount without counting) the number of dots on two 
separate cards and then use a method to find the sum. To find the sum, they may 
need to count the objects or use their fingers. 


You will need: Activity Description: Concept: 
O 5 sets of counting cards (0-5) In this game, students imagine that the cards Building early 
O 4 sets of numeral cards are brownies with chocolate chips and that a addition 
O Printed monster with slit cut in monster is eating the brownies. After the strategies 
mouth, placed over a container | students have determined the number of 
O Bowl or small trash can chocolate chips on two brownies, they write 
O 12 laminated blank numeral that number on a blank numeral card and 
cards ‘feed’ it to the monster. At the end of the 
game, the students can look inside and see 
what the monster ate. 


Early Addition and Subtraction 


Red 
(Counting Cards 


1 to 5) + (Counting 
Cards 0 to 5) 


Student models addition Student models addition Student can count on to add sums 
problems by counting problems as previously and in the range of 20, which includes 
each set and joining them, | also models subtraction counting from a number other 
then counting from 1 to problems by getting the larger | than 1 and tracking the number 
find the total. set, counting to remove the of counts necessary (e.g. to do 7 
smaller set, then counting to + 4 the student may count “8, 9, 
determine what remains. 10, 11,” while using fingers to 
Student may count from one to | keep track of the counts). 
add two sets. 


Activity Set-Up 


student’s current level of ability and extra support or additional challenges to help the child 
move to the next level) and draw out student thinking about the mathematics. A full sample 
lesson is provided in Appendix A. 


Every fourth club session was a Game Day. During Game Day, students were given the 
opportunity to choose from three or four activities set out by the facilitator. Facilitators could 
choose from new activities designed just for Game Day or from main activities that students had 
played previously in a regular club session. Game Days provided students with an opportunity 
to make choices, added variety, and allowed students to interact with materials independently. 
They also provided facilitators with an opportunity to reinforce concepts on which students 
needed more practice and to work with students who needed individualized instruction. 


A detailed pacing guide (see Appendix B) was provided to facilitators to prepare for 
each club session. Most clubs ran from October 2015 through May 2016. 


Communication with Families and Teachers 


Efforts were made to communicate with families and teachers throughout the year. Family 
Notes (in English and Spanish) were sent home regularly before each Game Day. The notes 
were designed to give families an overview of what was happening in clubs and ideas for 
activities to try at home. Twice over the course of the year, facilitators sent a personalized note 
to each student’s family that included one positive comment about the student’s behavior or 
participation in the group and one positive comment about what the student did mathematically 
during clubs. During the second half of the year, families were also provided with access to the 
Building Blocks online math practice program, which was designed to offer additional practice 
for students. Families were encouraged to log on and use the activities with their children. 


In addition, facilitators were twice given a note to distribute to their students’ classroom 
teachers. This note described what was happening in the clubs and offered the teacher an 
opportunity to meet in person with the facilitator. Few teachers took advantage of the opportuni- 
ty to meet with facilitators. 


The Learning Trajectory Framework 


To ensure that the High 5s curriculum would build on what students had learned in Making 
Pre-K Count, the curriculum for High 5s was developed using the learning trajectories 
research on which the Building Blocks curriculum is based. Learning trajectories are the 
developmental progressions through which children learn mathematics. Each trajectory 
includes a mathematical goal, a developmental path along which children develop on the way 
to achieving that goal, and a set of instructional activities that can help children develop 


higher levels of thinking as they progress toward that goal.' Figure 2.2 shows the counting 
trajectory that was used in High 5s. The High 5s activities were designed to move students 
along four different mathematical learning trajectories: counting, composition of numbers, 
early addition and subtraction, and geometry. The clubs also included some measurement and 
patterning activities. Main activities contained recommendations about how to tailor or 
differentiate the content for students at different points on the learning trajectory. The pacing 
of the program was intended to align, roughly, with the Go Math! curriculum used in most 
New York City kindergarten classrooms so that, for example, activities focused on counting 
were scheduled during the portion of the year in which counting was being emphasized in the 
classroom, and addition and subtraction activities occurred around the time that students were 
introduced to addition and subtraction in the classroom. However, the material in High 5s was 
designed to be somewhat more advanced than what students would have received in their 


Figure 2.2 
High 5s Counting Trajectory 


Counting 


Student counts 
from 1 while 


Counts 
accurately to 


Student counts 
by 10s and 1s 


Student counts 
by 1s fluently 


Student counts 
backward from 


Student counts 
verbally and 


pulling objects | with objects 10 and skip- through the (e.g. 10, 20, 30, | 200 and 

from a pile to | starting from a | counts by 10s_ | decade AO, 41, 42, 43) | beyond, 

‘get’ a given |number other | up to and transitions when counting recognizing the 

amount than 1 (e.g. beyond 100 starting at any | four sets of 10, | patterns of 1s, 

(progressing | gets 3 and verbally and number (e.g. asinabundle | 10s, and 100s 

from a small says ‘3’ then with object, 37, 38, 39, 40, | of sticks and (e.g. 197, 198, 

set to sets as | pulls another | such as bundles | 41). three single 199, 200, 201, 

large as 30). | and says ‘4’ of sticks or full objects. 202). Before 

and so on). ten frames. this, students 

often confuse 
what should 
come just after 
199 or 200. 


‘Clements and Sarama (2014). 


classrooms. For example, the counting activities in High 5s emphasized counting backward 
and skip counting (such as counting forward by 10s), while the kindergarten Go Math! 
curriculum typically only focused on counting forward by 1s. 


The research team developed each activity with the goal of supporting student learn- 
ing and development along the trajectory. The lessons were written anticipating that students 
would enter at one of the first levels, labeled red and orange, at the beginning of the year, with 
the goal of moving students toward the green and blue levels by the end of the year. Activities 
were written to support students at an initial level but also to provide opportunities for the 
facilitator to help students develop more advanced skills. In the addition activity “Chocolate 
Chip Count” (Figure 2.1), which was introduced on Day 6, students imagine that cards are 
brownies with chocolate chips and that a monster is eating the brownies. After the students 
have determined the number of chocolate chips on two brownies, they write that number on a 
blank numeral card and “feed” it to the monster. At the end of the game, the students can look 
inside and see what the monster ate. For students in red on the trajectory, the facilitator can 
use pairs of counting cards (which show both the numeral and the corresponding number of 
dots in a ten frame) in the ranges of | to 5 and 0 to 5 for the two brownies. For students in 
orange, the facilitator can use numeral cards (a card with just a numeral displayed) in the 
range of | to 9 and counting cards (0 to 5). And for students in yellow, the facilitator can use a 
pair of numeral cards (1 to 20 and 0 to 5). In each case, students are just adding 0 to 5 to the 
original addend, but it becomes more developmentally challenging as they move away from a 
visual representation of the number to the more abstract numeral. 


Formative Assessments 


To help facilitators understand each student’s place on the learning trajectory, individual 
formative assessments — which measure student progress and allow teachers to modify their 
approaches — were developed based on the learning trajectories in the areas of counting and 
early addition and subtraction. Students were asked to complete a small set of counting or 
addition and subtraction tasks similar to some of the High 5s activities that spanned a range of 
developmental ability. These assessments provided an opportunity for facilitators to sit with 
each student to learn what the student was able to do unassisted. Facilitators conducted assess- 
ments on scheduled Game Days. 


Summary 


In the High 5s program, facilitators met with small groups of students three times a week and 
used a clearly specified curriculum to play fun, engaging games that were designed to move 
students along mathematical learning trajectories in four different domains. Behind the 


scenes, a good deal of logistical and training support was put into place to ensure that the 
program would be implemented with fidelity and quality. This support is described in more 
detail in the next chapter. 
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Chapter 3 


Staffing the High 5s Program 


The High 5s program was implemented by Bank Street College of Education, which hired, 
trained, and supervised the facilitators. A total of 24 facilitators (most with bachelor’s degrees 
but limited formal teaching experience) administered the program to over 300 children in 24 
New York City public schools during the 2015-2016 school year. The facilitators were support- 
ed by five supervisors and a program director at Bank Street College. Bank Street also provided 
administrative, information technology, and human resources support for the program. 


As shown in Table 3.1, the group of facilitators hired for the project was diverse. They 
ranged in age from 22 to 39 and came from a variety of racial and ethnic backgrounds. A 
majority were recent college graduates who had an interest in education or related course work. 
They were mostly female (75 percent), and 29 percent identified as Hispanic or Latino, 38 
percent as white, 29 percent as black, and 4 percent as some other race. One-quarter were fluent 
in Spanish and 83 percent had a bachelor’s degree. Just over a quarter of the facilitators had 
their highest degree in the field of education, but none had a New York State teaching certifi- 
cate. They averaged less than two years of formal teaching experience, defined as an “assistant 
or lead teacher, including student teaching’; about one-third (30 percent) had none. Most had 
some experience working with children, defined as experience in “fa non-academic or non- 
classroom setting (e.g., tutoring, summer camp, after school program).” Facilitators were paid a 
salary commensurate with that of a paraprofessional teacher in the New York City public 
schools (around $25 per hour, depending on experience). 


The hiring process was extensive; facilitators were recruited through a variety of meth- 
ods. A job description was posted at Bank Street College, at colleges and universities with early 
childhood programs throughout New York City and New Jersey, and on a number of social 
media and job search outlets such as the Day Care Council of New York, Indeed, and the 
National Association for the Education of Young Children. Bank Street and MDRC also 
partnered with the City University of New York and held recruiting events at Brooklyn and 
Lehman Colleges. Over 300 résumés were received, 107 candidates were offered a phone 
interview, and 48 candidates were interviewed in person. Ultimately 24 facilitators were hired. 


Training and Supervision 


The High 5s model involved a substantial amount of training and supervision designed to 
support facilitators over the course of the year. The original study design called for only a few 
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Table 3.1 


Facilitator Demographics 


Characteristic 
Race/ethnicity (%) 
Hispanic 29.2 
Non-Hispanic white 37.5 
Non-Hispanic black 29.2 
Other/multiracial* 4.2 
Bachelor's degree (%) 83.3 
Degree in education (%) 26.1 
Female (%) 75.0 
Fluent in Spanish (%) 25.0 
Age? (mean) 26.3 
Years of formal teaching experience® (mean) 1.6 
Years of informal experience working with kids® (mean) 4.0 
Facilitators 24 


SOURCE: MDRC calculations based on a demographic survey 
administered to High 5s club facilitators in spring 2016. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

aOther/multiracial includes Asian, Native Hawaiian/Pacific Islander, and 
American Indian/Alaska Native; facilitators who identified as the option 
"some other race"; and facilitators who selected more than one race on the 
survey. 

‘Facilitator age ranged from 22 through 39 years old. 

°Facilitator has experience as an assistant or lead teacher, including 
student teaching for all grades and preschool. Formal teaching experience 
ranged from 0 to 5.8 years. 

dFacilitator has experience working with children in a nonacademic or 
nonclassroom setting (e.g., tutoring, summer camp, after-school program). 
Informal experience working with kids ranged from 0 to 14 years. 


days of training, but based on a pilot conducted during the 2014-2015 school year, it was 
decided that facilitators would need more training to implement the program with fidelity to the 
model. This was particularly true given the diverse backgrounds of the facilitators. Those 
without formal classroom experience or a mathematics education background needed additional 
training in the mathematical concepts behind the curriculum, small-group management, and 
using high-quality instructional practices (for example, differentiated instruction and encourag- 
ing mathematical thinking). 
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Supervisors began training on August 3, 2015, and received 18 days of training from 
the program director and research team during that month. They also spent time preparing 
materials and planning for facilitator training. The program director and three of the five 
supervisors had been Making Pre-K Count coaches and were already familiar with the Building 
Blocks instructional approach, which helped them get up to speed quickly on the High 5s 
curriculum. All the supervisors had significant experience working in early childhood education 
as well. Facilitators received 16 days of training in August and September (about 6 of which 
were spent on research-related activities, such as learning to log information about their clubs 
for research purposes, or administrative tasks, like technology setup and fingerprinting). 


Training — conducted by staff members from Bank Street, MDRC, and the University 
of Michigan, as well as Doug Clements, the codeveloper of Building Blocks — incorporated a 
variety of topics, including how to teach mathematics in an age-appropriate manner and how to 
facilitate small groups, as well as practice implementing the curricular activities. Facilitators 
were trained on mathematical learning trajectories at the beginning of the sequence. Facilitators 
were also given an opportunity to observe in a pre-K or kindergarten classroom. 


All facilitators completed a certification process at the end of training. Facilitators were 
asked to conduct a short version of a club (one start-up and the main activity) from beginning to 
end. To be certified, facilitators had to meet the time requirements and receive a 3 or above (on 
a 5-point scale) for all the instructional quality and fidelity items. 


Eight additional days of training at Bank Street College were conducted by the research 
team, program director, and supervisors throughout the school year. These training sessions 
were held on days when the public schools were closed for in-service or school breaks. Topics 
of training sessions held during the year were chosen in response to the needs of the facilitators 
as expressed to supervisors and reflected in management information system data. These 
training sessions also gave facilitators an opportunity to build a professional community and 
learn from each other. 


Supervision Model 


Supervisors met frequently with facilitators throughout the year, providing both logistical and 
instructional support. Each supervisor was responsible for a cohort of four to five High 5s 
facilitators and met with the cohort weekly. Supervisors also met with facilitators individually 
as needed — they were intended to meet, at a minimum, once every few weeks — and provided 
coaching in the field as necessary. Supervisors reviewed data from the daily facilitator logs 
frequently to monitor timely completion of logs, completion of activities as scheduled, curricu- 
lar challenges, behavioral or attendance issues, and logistical problems. Facilitators communi- 
cated by email, phone, or text with supervisors as needed. 
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Cohort Meetings 


Members of the research team observed a weekly meeting of each cohort in April 2016. 
Supervisors were free to structure their meetings to best fit the needs of their facilitators, but all 
meetings included some combination of the following activities: logistics, curriculum review, 
reflection about students, and professional development. Logistical topics included reminders 
about school closings, when to introduce the Building Blocks software, and space issues. 


Curriculum review varied in format. In some cohorts, a facilitator was assigned an ac- 
tivity to prepare and practice with the group. In others, the group discussed a set of activities as 
a group and then had time to practice them individually or in pairs. Each group also spent time 
reflecting on their students. In one cohort, all facilitators prepared case studies on one or two 
students, and one facilitator was selected to share his or her case study with the group. The 
group brainstormed ideas to help the student in question and facilitators were encouraged to 
think about how the ideas generated could be applied to other students. In another cohort, 
facilitators wrote notes on large sheets of paper about how their students engaged with the set of 
activities from the previous week and then discussed each sheet as a group. Discussions 
included tactics for differentiating and scaffolding students at different places on the learning 
trajectories. Finally, cohort meetings also included some type of professional development, 
including discussion of instructional practices, mathematics research, or group management 
strategies. At the end of the year, one facilitator wrote: 


My supervisor did a great job of creating an environment where we felt comfort- 
able sharing our challenges with each other in our meetings. Our cohort meetings 
were so valuable that our relationships extended outside of them. I frequently 
spoke with and presented challenges to members of my cohort out in the field on 
a daily basis. I felt like I was being mentored by every member of my cohort and 
I don’t think this experience would have been the same without them. 


Individual Coaching and Supervision 


Individual supervision involved both big-picture topics and issues specific to the facili- 
tator. Supervisors also viewed individual supervision as a time to provide oversight and ensure 
that facilitators were accountable for their work. The frequency of individual meetings changed 
during the year and depended on need, but they typically occurred once every two to three 
weeks. 


Supervisors also provided coaching in the field to help facilitators integrate what they 
learned from training and supervision and apply it to their clubs. Coaching served the dual 
purpose of supporting facilitators in addressing challenges in their clubs and helping to improve 
the quality of implementation. All facilitators were observed and given coaching in the field at 
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least every other month. Some facilitators with particularly challenging clubs or less experience 
working with young children received coaching as often as once a week. 


In general, the staff hiring, training, and supervision model was deemed effective. The 
hiring process went smoothly and resulted in a qualified and committed set of supervisors and 
facilitators. The training and supervision model was well received. Most facilitators indicated 
that they felt prepared for their roles and that they received enough supervision. Nineteen of 
24 indicated that the amount of supervision they received was “just right”; four said there was 
too much supervision. Facilitators indicated that they felt like part of a team, and retention 
was high. No facilitators resigned because they were dissatisfied with the position; two took 
time off for maternity leave and one left due to a personal issue. At the end of the year, one 
facilitator wrote: 


I think one thing that made this job really stand out from any other job I’ve had 
was how much support I had as a facilitator. I honestly felt like I could count on 
every single person in our team to help me and that made me feel more secure in 
what I was doing. 


Logistical Support for the Program 


The High 5s program, as implemented in this study, also involved substantial logistical 
support from MDRC and the Bank Street supervisors, particularly regarding scheduling and 
attendance. At the beginning of the year, MDRC staff members worked closely with schools to 
find a convenient time and an appropriate place to hold clubs. In many schools, both the timing 
and location of clubs had to be rescheduled throughout the year to accommodate changes in 
school schedules. MDRC staff members, Bank Street supervisors, and school staff members 
worked hard to ensure that there would be an appropriate time and place for clubs to meet. 
MDRC and Bank Street also followed up regularly with families when children were absent and 
worked to find solutions to chronic absenteeism, particularly at the beginning of the year. 


Summary 


The High 5s facilitators who were hired for the project were a responsible and dedicated group, 
but for the most part they came to the program with limited formal teaching experience. An 
extensive array of training and support structures were put in place to help them implement the 
High 5s program with fidelity, including not only initial training but also weekly supervision 
delivered in a variety of forms. As will be described in more detail in the next chapter, this 
support may have helped to ensure that the implementation of the program was strong. 
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Chapter 4 


Quality and Fidelity of Implementation 


When the project team set out to design a math enrichment program that would be provided to 
children outside of regular instructional time, it remained to be seen whether schools would be 
receptive, whether enough facilitators could be hired at a paraprofessional-level salary and 
retained over the course of an entire school year, whether an appropriate time and place to hold 
the clubs could be identified, and whether children would attend regularly and be engaged. But 
as described in Chapter 3, facilitators were indeed hired and retained, and with the help of a 
strong training and supervision model, the implementation of the High 5s program was success- 
ful. The program met all the benchmarks identified at the outset: Throughout the year, attend- 
ance and engagement were both high, sessions were held on a regular basis, and activities were 
implemented with fidelity. 


In assessing the implementation of the High 5s program, the team set out to answer the 
following questions: 


1. Did children attend regularly (at least 80 percent of the time)? 


2. Did facilitators implement the clubs with fidelity to the program model? Specifi- 
cally, did they spend at least 20 minutes per club on math activities, implement 
the correct activities on the correct days, deliver at least one start-up and the main 
activity in each club, and set up materials correctly and conduct the activities as 
intended? 


3. What was the overall quality of instruction? Specifically, did facilitators develop a 
positive instructional climate, encourage mathematical thinking, and differentiate 
instruction for students at different points on the learning trajectory? 


4. Were students engaged during the clubs? 
5. What logistical challenges did the program face? 


6. How was the High 5s program received by teachers and school administrators? 
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Implementation Data 


Program implementation was measured using a variety of methods: logs, observations, and 
facilitator reflections, as well as surveys and interviews. First, facilitators were asked to com- 
plete logs after every club session.' Supervisors closely monitored the log data to ensure that 
facilitators were filling out logs for each club. Compliance was high, and all logs were filled out 
in a timely manner. Over 8,000 club logs were completed across the school year. 


The research team also conducted observations of clubs. High 5s clubs were observed 
by trained MDRC staff members and outside consultants in both the fall and spring of the 
program year.” Observers received training and were certified on the observational instruments 
before they were able to conduct observations in the field. Each facilitator was observed once in 
each semester, for a total of 25 observations in the fall and 26 in the spring; at least one club in 
each school was observed.* The particular clubs that were observed in the fall were not neces- 
sarily observed again in the spring, although many were. 


Facilitators completed exit memos at the end of the program in which they reflected on 
their experience. Finally, focus groups, surveys, and interviews were conducted with school 
administrators. The data collected from these sources provide the information to assess the 
following measures of program implementation: 


e Child attendance. The logs asked facilitators to indicate which children 
were present, absent, or in partial attendance (that is, came late or left early 
but were there for at least 10 minutes) each day. 


e Fidelity to the High 5s model. To measure the degree to which facilitators 
were implementing the program with fidelity, the logs included items in 
which facilitators recorded the activities that were implemented each day, the 
amount of time spent on activities, and, if the club was canceled for any rea- 
son, the reason why. The research team that conducted observations also re- 
corded information about the activities that were implemented and the length 
of those activities. In addition, researchers rated facilitators on the degree to 


'See Appendix C for the full log content. 

*Fall observations took place in November 2015. Spring observations took place in March 2016, except 
for four observations on the last day of February 2016. 

3In order to observe in all schools and to observe each of the facilitators, it was necessary to conduct 
more than 24 observations. In the fall, this meant that one facilitator was observed twice. In the spring, 
one facilitator was on maternity leave and one had left the project for personal reasons. To cover the clubs 
originally run by these two facilitators, one full-time and two substitute facilitators were hired, and all 25 
of the facilitators were observed. Again, one facilitator was observed twice so that all schools could be 
included in the observations. 


18 


which the activities were conducted as written, materials were set up correct- 
ly, and facilitators displayed an understanding of the math behind the activi- 
ties they were implementing. Ratings were given on a 5-point scale where a 
rating of 3 indicated meeting expectations and 4 or above indicated exceed- 
ing expectations. 


Instructional quality. Measures of the quality of instruction were drawn 
mainly from club observations. Observers rated facilitators on a number of 
dimensions, including the instructional climate of the clubs, the amount of 
mathematical reflection that facilitators engaged in with their students, and 
the degree to which facilitators differentiated instruction for children at dif- 
ferent points on the learning trajectory. Ratings were given on a 5-point scale 
where a 3 indicated meeting expectations and 4 or above indicated exceeding 
expectations. In addition, facilitator exit memos illuminated the instructional 
areas that facilitators found most challenging over the course of the year. 


Student engagement. Daily logs also included items about child engage- 
ment, which asked facilitators to rate the level of engagement for both the 
start-up activities and the main activity on a scale of | (all students are disen- 
gaged) to 5 (all students are engaged). During the pilot, student engagement 
numbers were used to modify activities or the pacing guide if necessary, but 
during full-scale implementation they were used primarily to guide supervi- 
sors in supporting facilitators. During club observations in the fall and spring, 
researchers also rated the level of student engagement using the same rating 
scale. During spring observations only, the research staff recorded the degree 
to which students were engaged with one another and the number of students 
in each club who were consistently engaged throughout the session. 


Logistics. Facilitators recorded any challenges they faced each day on their 
daily logs, and observers also recorded any logistical difficulties they ob- 
served. Facilitators reflected on the challenges they faced more generally 
across the school year in their exit memos. 


School leaders’ perceptions of the program. Information regarding teacher 
and principal perceptions of the program were taken from focus groups, sur- 
veys, and interviews conducted with school administrators, as well as from 
facilitator exit memos. 
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Implementation Findings 


Child Attendance 


Student attendance was high. At the outset of the program, a benchmark was set for 
overall student attendance rates of at least 80 percent. That benchmark was exceeded; the 
average attendance was 87.4 percent over the year. As shown in Figure 4.1, four out of five 
students had attendance rates over 80 percent, and nearly all students (more than 90 percent) 
had attendance rates over 70 percent. To achieve this rate, supervisors monitored attendance; if 
a student was absent or had partial attendance two or more times over a two-week period, 
supervisors would follow up with the facilitator. If necessary, someone from the project team 
would contact the student’s parent or guardian. The high attendance rate was a result of signifi- 
cant time and energy devoted by both MDRC staff members and Bank Street College supervi- 
sors to contacting families whose children were absent from clubs and making scheduling and 
other changes to accommodate each family’s needs. 


Figure 4.1 
High 5s Club Attendance Rates 


More than 
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144 (50.7% of sample) | children had 
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rate of at least 
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SOURCE: MDRC calculations based on High 5s facilitators’ club session logs. 


NOTE: Club attendance was calculated for each child who did not drop out of High 5s by dividing the number of 
club sessions for which a student was present or partially present by the total number of sessions held. 
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Fidelity to the High 5s Model 


As noted earlier, each club included two start-up activities that were intended to last 7 to 
10 minutes and one main activity that was intended to last 15 to 20 minutes. The total time 
spent on mathematics content was intended to be greater than 20 minutes each session. As 
shown in Table 4.1, facilitators consistently met these benchmarks. On average, clubs spent 
almost 25 minutes on math during a session. Facilitators spent an average of 8 to 9 minutes on 
start-up activities and 16 to 17 minutes on main activities. Club observations confirmed facilita- 
tors’ reports. 


Club sessions were completed as scheduled, and the curriculum was delivered as in- 
tended. As shown in Table 4.2, according to facilitator logs, 93 percent of scheduled sessions 
were completed. Sessions were canceled primarily due to changes in school schedules, unavail- 
ability of space, or an insufficient number of students in attendance.‘ Facilitators also reported 


Table 4.1 


Amount of Time Spent on Math in High 5s Clubs, 
Based on Facilitator Logs and Spring Club Observations 


Facilitator Logs Spring Club Observations 

Standard Standard 

Use of Time Mean Deviation Mean _ Deviation 

Minutes spent on start-up activities 8.4 2.26 8.5 2.42 

Minutes spent on main activities 16.5 3.03 15.6 2.95 

Minutes spent on math? 24.7 4.17 24.1 4.11 

Sample size 

Sites 24 23 
Clubs 79 26 
Club sessions” 5,971 26 


SOURCES: MDRC calculations based on facilitators' club session logs and on observations 
of individual clubs conducted in spring 2016 using an instrument developed by MDRC 
research staff to assess implementation fidelity. 


NOTES: 2Minutes spent on math includes time spent on start-up and main activities. 

>The number of club sessions includes only those with scheduled activities and does not 
include Game Days, when students were given the opportunity to choose from four activities 
set out by the facilitator. During observation, only one session of each club was observed, so 
the number of sessions observed is the same as the number of clubs. 


4Clubs were held if two or more students were in attendance. 
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Table 4.2 


Fidelity to the Model: Instructional Quality, 


Based on Facilitator Logs and Club Observations 


Measures of Fidelity (%) 


Scheduled sessions held? 
Completed at least one start-up 
Completed both start-ups 
Completed main activity 
Correct activities conducted 


Facilitator met or exceeded expectations? 
Conducted activities as written 
Set up materials correctly/was familiar with the activity 
Displayed an understanding of mathematical concepts, 
using correct vocabulary, and making no significant 
mathematical mistakes 


Sample size 
Sites 
Club sessions® 


SOURCES: MDRC calculations based on facilitators’ club session logs and on observations of 


Facilitator Logs 


92.5 
98.6 
89.9 
99.6 
96.2 


24 
7,031 


Club Observations 


Fall 


100.0 


24 
25 


Spring 


100.0 
92.3 
100.0 
92.3 


92.3 
100.0 


96.2 


23 
26 


individual club sessions conducted in fall 2015 and in spring 2016 using an instrument developed 


by MDRC research staff to assess implementation fidelity. 


NOTES: #Percentage of scheduled sessions held is calculated by dividing the number of clubs held 


by the total number of expected sessions. 


Facilitators were rated on a 1 to 5 scale. Facilitators who met or exceeded expectations 


received a rating of 3 or above. 


‘For facilitator logs, the sample size for club sessions is the total number of expected sessions. 
The denominator for the variables related to implementation fidelity from the facilitator logs includes 


only sessions with scheduled activities (5,971), not Game Days, when students were given the 


opportunity to choose from four activities set out by the facilitator. 


As shown in Table 4.2, club observations also indicated that activities were conducted 
as written, materials were set up correctly and facilitators were familiar with the activities they 
were implementing, and facilitators understood the mathematical concepts they were teaching. 
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that the main activity was completed in more than 99 percent of the completed sessions and that 
both start-ups were conducted in 90 percent of sessions (at least one start-up was conducted in 
99 percent of sessions). Facilitators conducted the specific activities that were scheduled for that 
day in 96 percent of clubs. These findings were corroborated in club observations. On average, 
82 of the 89 sessions that were developed and included in the pacing guide were completed, 
with a range of 66 to 89 sessions completed (not shown in table). 


Respectively, 96 percent, 88 percent, and 100 percent of the facilitators met expectations in 
these areas in the fall, and similar numbers were observed in the spring. 


Instructional Quality 


While the program was implemented in a fun, game-like atmosphere with fidelity to the 
model, ensuring the highest quality of instruction — instruction that included differentiation, 
involved reflective questioning, and underscored the mathematical objective of the activities — 
was challenging, as it is for many teachers, despite ongoing and high-quality training and 
coaching. 


To provide an overall perspective on the quality of instruction, the study team looked 
at average facilitator quality ratings over the course of the school year. Averaging across all 
12 instructional quality ratings given by observers in both the fall and spring, six facilitators 
were rated consistently high on all categories (that is, their average rating was 4 or above on a 
5-point scale where 4 represents exceeding expectations). Five facilitators had average scores 
below 3 on a 5-point scale on which 3 was considered satisfactory. The remaining 14 facilita- 
tors had average scores above 3 but below 4, indicating that they met expectations but did not 
exceed them. 


Figure 4.2 shows a more detailed breakdown of the ratings of facilitators’ instructional 
practice during the observations that were conducted in the spring. Facilitators were rated on the 
instructional climate of the clubs, the degree to which they encouraged mathematical thinking 
among their students, and how well they differentiated instruction. The table shows the percent- 
age of facilitators who were deemed to be exceeding expectations for each item (a rating of 4 or 
5 on a 5-point scale) and the percentage who either met or exceeded expectations (a rating of 3 
or higher on a 5-point scale). 


Facilitators created a positive instructional climate for children. All facilitators met ex- 
pectations for having good rapport with students and avoiding negative strategies to manage 
behavior, and over 80 percent met expectations for making math learning fun and using positive 
strategies to manage behavior. Many facilitators consistently encouraged effort and persistence 
over getting the right answer, with 73 percent meeting expectations and 39 percent exceeding 
expectations. The instructional climate that was developed in the clubs is illustrated by the 
following quotes from facilitators when asked to reflect at the end of the year on the most 
rewarding aspect of their job: 


Watching my kids understand the concepts and retain strategies for future use. 
When I saw their faces light up as they arrived to the answer on their own. 
Growing my relationships with my kids and them knowing they can make a mis- 
take without it being a problem. 
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Figure 4.2 


Facilitators Meeting and Exceeding Expectations of Implementation Quality 


Instructional climate 
Has good rapport with students 


Avoids negative strategies 
Makes math learning fun 
Uses positive classroom management strategies 


Encourages effort and persistence over right answer 


Encouraging mathematical 
eee Draws attention to math 


thinking 
Underscores the mathematical objective 
Encourages mathematical reflection 
Asks open-ended questions 
Differentiated 
instruction Provides opportunity for all to participate 


Scaffolds main activity 


Scaffolds start-up activity 


16.9% Eo 7 100.0% 
16.9% a 100.0% 
53.9% _——s=~"| 88.5% 


65.4% 84.6% 
38.5% 73.1% 


=== 02.9% 
== 27 EE 02.9% 
37 80.2% 
30S 50.0% 


65.4% 


84.6% 
73.1% 


26.9% 


23.1% 61.5% 


m= Meeting or exceeding expectations 
Exceeding expectations 


SOURCE: MDRC calculations based on observations of 26 High 5s clubs conducted in spring 2016 using an 
instrument developed by MDRC research staff to assess implementation fidelity of club activities. 


NOTE: Facilitators were rated on a 1 to 5 scale. Facilitators who met or exceeded expectations received a 
rating of 3 or above. Facilitators who exceeded expectations received a rating of 4 or above. 


Forming relationships with my students, and seeing them grow over the course 
of the year. One student stands out in particular — she was often frustrated at 
the beginning of the year because she could not get to the answers or use the 
strategies to solve problems as easily as her peers. Now, she easily counts on, 
counts by 10s past 200, and knows how to apply addition strategies to different 
questions. Not only does she have all of these skills but also her confidence is 


striking to me. 


As can be seen in Figure 4.2, encouraging mathematical thinking and differentiating in- 


struction were somewhat more difficult for facilitators. The Building Blocks curriculum on 
which High 5s was based emphasizes asking open-ended questions and helping students engage 
in mathematical reflection. In the spring observations, facilitators were given a rating of 4 or 5 
on an item asking about open-ended questions in 31 percent of the observations (where 5 
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indicates that the facilitator asked open-ended questions throughout the lesson) and were rated 
as meeting expectations in 50 percent of the observations. In 35 percent of the observations, 
facilitators were rated a 4 or 5 on an item asking about mathematical reflection (where 5 
indicates that the facilitator reviewed key ideas for all activities). Throughout the year, facilita- 
tors were encouraged to underscore the mathematical objective of each activity during club 
sessions. In 46 percent of the spring club observations facilitators were rated as exceeding 
expectations on this item. 


Options for differentiation were written into the curriculum, were provided in supervi- 
sion, and were a focus of training and supervision in the second half of the year. In the spring 
club observations, 73 percent met expectations regarding scaffolding the main activity (giving 
individual children extra support or challenges when needed to help them move to the next skill 
level), while 27 percent of facilitators exceeded expectations on this item. These practices are 
difficult even for more experienced classroom teachers, as will be illustrated in findings from 
the kindergarten classroom observations described in Chapter 5. 


Another challenge experienced by some clubs was managing the behavior of small 
groups of young children. Facilitators received training in small-group management and 
establishing behavioral expectations before entering the field, but some felt they were unpre- 
pared for the level of challenge they encountered. In exit interviews, about one-third of the 
facilitators (7 of 24) listed difficult behaviors or group dynamics as the most challenging aspect 
of their job. In approximately 14 percent of the clubs, facilitators spent substantial time manag- 
ing behavior and, as a result, were less able to focus on the mathematics or on other aspects of 
their instructional delivery.° Supervisors provided ongoing support to clubs experiencing high 
levels of challenging behavior throughout the year, but this took time away from other instruc- 
tional support activities in which they might have been engaged. As noted above, however, 
facilitators were able to avoid negative strategies for managing behavior, probably because of 
the training and supervision they received. 


Student Engagement 


As shown in Table 4.3, student engagement was also high (as reported by both facilita- 
tors and observers). Engagement was rated on a scale of | (all students are disengaged) to 5 (all 
students are engaged). As rated by facilitators, nearly all children were engaged in most clubs, 


‘Facilitators indicated in their daily logs the degree to which they experienced challenging behavior in 
each club. Over the course of the entire year, four clubs had an average rating above 2.0 (where 1 = no 
instances of challenging behavior, 2 = intermittent minor challenges, 3 = frequent minor challenges or 
intermittent intense challenges, 4 = somewhat frequent and intense challenges, and 5 = ongoing and intense 
challenges). Another seven clubs had a yearly average between 1.7 and 2.0. 
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Table 4.3 


Student Engagement, 
Based on Facilitator Logs and Spring Club Observations 


Facilitator Logs Spring Club Observations 

Standard Standard 

Activity Mean Deviation Mean Deviation 

Start-up activities 4.8 0.55 4.0 0.66 

Main activity 4.7 0.65 3.9 0.74 

Game Days 4.9 0.36 - - 

Engagement with other children* - - 2.7 0.98 

Sample size 

Sites 24 23 
Clubs 79 26 
Club sessions? 5,971 26 


SOURCES: MDRC calculations based on facilitators’ club session logs and on observations of individual 
clubs conducted in spring 2016 using an instrument developed by MDRC research staff to assess 
implementation fidelity. 


NOTES: Engagement was rated on a scale of 1 (all students are disengaged) to 5 (all students are 
engaged). 

aObservers rated the extent to which "students are responsive to and engage with one another while 
working on activities with or without the support of the facilitator." 

>The sample size for the start-up and main activity engagement items from the facilitator logs includes 
only sessions with scheduled activities, not Game Days. The sample size for Game Day club sessions is 
1,749. During club observations, only one session of each club was observed, so the number of sessions 
observed is the same as the number of clubs. 


with average scores of 4.8 for start-up activities, 4.7 for main activities, and 4.9 for Game Days. 
Club observations confirmed high levels of engagement, although not quite as high: 4.0 for the 
start-ups and 3.9 for the main activities in the spring observations.° Children’s engagement with 
other children was somewhat lower (observers rated this as 2.7 on average). The number of 
children consistently on task averaged a little more than two (not shown), meaning that while 
students were generally engaged, it was difficult to keep them consistently on task throughout 
the entire club session. 


Logistics 


As expected, the detailed logistics of running the program proved to be one of the 
most time-consuming aspects. Getting students to and from the club was the most frequently 


°In the fall, observers did not rate engagement separately for start-ups and main activities; instead they 
rated engagement “throughout the lesson.” In the fall average engagement was 3.7 on the 5-point scale. 
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cited problem in the facilitator logs and in club observations. Facilitators picked up students 
from the lunch room or their classrooms, but changes in the school schedule or teacher 
absences often caused students to be in different locations from day to day and week to week. 
Similarly, several sites experienced recurring issues with finding an appropriate space for 
clubs to meet. It took substantial staff time from MDRC, Bank Street, and the schools to work 
around these issues. 


No ideal time to hold clubs was identified; each possible time frame had strengths and 
weaknesses. After-school clubs experienced the fewest problems with attendance and finding 
and keeping a dedicated space to meet, and offered the most time for completing activities. 
Facilitators reported, however, that students were often tired or emotionally spent at the end of 
the day. Before-school clubs faced the biggest challenges in terms of child attendance. Lunch 
clubs had fewer attendance problems than before-school clubs, but facilitators reported that they 
had difficulty locating children in the lunch room and frequently felt rushed. Sometimes 
students who attended lunchtime clubs missed part of recess to attend, which was not ideal. 
Commuting between schools was also difficult for facilitators. In exit interviews, 7 of the 24 
facilitators listed the commute as the biggest challenge they encountered as a facilitator. 


As described in Chapter 6, many of these logistical issues might have been mitigated by 
integration of the program staff into the school, which might be more feasible in a non-research- 
based implementation of High 5s. However, as noted in Chapter 1, operating the program 
independently gave the study team maximum flexibility in hiring and training the staff and thus 
increased the chances that the program would be implemented with fidelity right from the start. 
This was particularly important because program impacts were measured at the end of the first 
year of implementation. 


School Leaders’ Perceptions of the Program 


The program was generally well received by the schools. When asked to rate the sup- 
port they received from their schools for the program, facilitators cited only three schools in 
which they felt unsupported by the school administration and staff. The backing for the program 
may have been due, in part, to the strong relationship that was developed with these schools as 
part of the larger Making Pre-K Count project. During end-of-the-year focus groups, interviews, 
and surveys, most administrators confirmed this robust support. On the surveys, administrators 
generally agreed or strongly agreed that they were satisfied with High 5s, with a mean of 4.6 on 
a scale of 1 (strongly disagree) to 5 (strongly agree). In interviews and focus groups, most 
administrators mentioned at least one of the following as a benefit of High 5s for the children 
who participated: extra math support, enjoyable math lessons, more time for small-group work, 
improved math skills, and a sense of structure and independence. 
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Summary 


Overall, the implementation of the High 5s program went as intended and met the expectations 
that had been established at the outset. This was encouraging because High 5s was a newly 
developed program, and initially it was unclear whether it could be implemented successfully. 
The training and supervision that were provided to facilitators may have contributed to the 
successful implementation. Having several supervisors who were already knowledgeable about 
the Building Blocks program also may have helped. In the concluding chapter of the report, 
considerations for replicating the model in other contexts are discussed. 
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Chapter 5 


Instruction in the Kindergarten Classrooms 
Compared with High 5s 


All students who were part of the High 5s study also received typical math instruction in their 
kindergarten classrooms. To better understand the value the High 5s program may add, the 
research team observed kindergarten teachers and studied the nature of the mathematics 
instruction students were receiving in their classrooms, compared with High 5s clubs. Instruc- 
tion in kindergarten classrooms was found to differ in a number of ways from the instruction in 
High 5s, both in the content covered and in the mode of delivery. Classroom instruction was 
primarily delivered in a whole-group format or with children working individually in their seats. 
In part because there was typically only one teacher in each classroom, there were few opportu- 
nities for hands-on, small-group experiences. 


Researchers observed kindergarten classrooms in the 24 High 5s schools in the fall and 
spring of the kindergarten year. The classrooms included students who participated in High 5s 
in kindergarten and received enriched math in prekindergarten (Making Pre-K Count and High 
5s), students who received enriched math in pre-K only (Making Pre-K Count only), and 
students who were not part of the study and had received neither Making Pre-K Count in pre-K 
nor High 5s in kindergarten. A total of 75 observations were conducted, with participating 
classrooms (n = 42) observed once in the fall and once in the spring. When a teacher who was 
observed in the fall was not available in the spring, a replacement teacher was selected for 
observation. ! 


Two types of observations were conducted: math block and full day. The math block 
observations were approximately 60 minutes long and were conducted during regularly sched- 
uled math instructional time. Full-day observations were intended to capture any mathematics 
instruction that occurred outside of the regular math lesson and to measure the amount of 
mathematics in relation to other subjects. Observers included MDRC staff members and 
researchers from the University of Chicago, the University of Michigan, and Vanderbilt 
University. 


The observation protocol included a simplified version of the Narrative Record and an 
adapted version of the Classroom Observation of Early Mathematics — Environment and 


'Thirty-three teachers were observed in both the fall and the spring. Three were observed only in the fall, 
and six new teachers were observed in the spring. 
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Teaching (COEMET).* The Narrative Record is an open-ended format for describing the types 
of activities and the content of instruction. The COEMET describes math instruction during 
formal math activities, including items related to math practices, type of activity, student 
engagement, and instructional quality. As with the club observations, kindergarten classroom 
observers received training and were certified on the observation instruments before they were 
able to conduct observations in the field. Several of the items in the observation protocol were 
identical to items that were included in the High 5s club observation protocol, which helps 
facilitate comparisons. About one in five observations were checked for reliability by having a 
master coder observe and code with the staff member conducting the observation. Overall 
reliability calculated for these double-coded sessions using Cohen’s kappa was 0.92. 


The Kindergarten Classrooms 


Table 5.1 provides descriptive statistics on the typical kindergarten classroom in these schools. 
Between 19 and 20 students were present, on average, in the observed classrooms. This is lower 
than the national and New York State primary school averages of 21.6 and 21.5, respectively.? 
Most classrooms included between one and eight High 5s program students and a similar 
number of Making Pre-K Count only students. Fewer than 20 percent of classrooms had a 
teacher’s assistant. A larger percentage did have another staff member or adult present in the 
classroom for at least 10 minutes. However, the role of these adults and the length of time spent 
in the classroom varied widely. The roles included, but were not limited to, grandparent or 
parent volunteers, teacher’s aides, and reading specialists. 


As shown in Figure 5.1, teachers spent 56 minutes on average per day on math instruc- 
tion. This compares with approximately 97 minutes per day spent on literacy instruction and 
almost 50 minutes spent on other academic content (including science, social studies, art, and 
music) combined. Most math instruction was conducted during the math block. Teachers spent 
fewer than 10 minutes on math, on average, outside of the math block. Forty-two percent of the 
day (more than two and a half hours) was spent in noninstructional transitions or meals. 


The Narrative Record was developed by Dale C. Farran and Carol Bilbrey at the Peabody Research Insti- 
tute, Vanderbilt University, Nashville (Farran and Bilbrey, 2004). The COEMET was developed by Julie 
Sarama and Douglas H. Clements at the State University of New York at Buffalo (Sarama and Clements, 
2007). 


National Center for Education Statistics (2013). 
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Table 5.1 


Description of the Kindergarten Classroom, 
Based on Classroom Observations 


Outcome Fall Spring 
Average number of students present in the classroom 19.6 19.2 
Classrooms with a teacher assistant present (%) 19.4 12.8 
Classrooms with another adult ever present* (%) 38.9 25.6 
Sample size 
Sites 21 24 
Classrooms 36 39 


SOURCES: MDRC calculations based on full-day and math block observations 
conducted in fall 2015 and spring 2016 using a version of the Classroom 
Observation of Early Mathematics—Environment and Teaching (COEMET; 
Sarama and Clements, 2007) that was modified for the High 5s study. Math 
block observations were approximately 60 minutes long and were conducted 
during regularly scheduled math instruction time. Full-day observations took 
place over the entire school day. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

4Another adult ever present is defined to include any aides or school staff 
members present for longer than 10 minutes during the observation and does 
not include the participating lead teacher or primary teacher assistant. 


Mathematics Instruction in the Kindergarten Classroom 


The mathematics portion of the observation protocol included information on the con- 
tent and organization of the math activities and the materials that were used, as well as items 
intended to capture information about instructional quality. 


As shown in Table 5.2, some aspects of the observed math instruction were strong. A 
majority of the math activities were delivered clearly, teachers made relatively few mathemati- 
cal errors, and teachers were actively involved in most activities (for example, teachers typically 
circulated during seat work and helped individual students, as opposed to sitting at their desks 
grading papers). 


Other aspects were less strong. Research has shown more student growth in mathemati- 
cal knowledge when teachers talk about math with their students.* To assess how much teachers 
helped students engage in mathematical thinking, observers rated each math activity on (1) the 


4Klibanoff et al. (2006). 
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Figure 5.1 


Instructional Content in Kindergarten Classrooms in Fall 2015 


None? Literature/Language 
153.1 min. 96.7 min. 


56.1 min. 


Gross Motor® 


10.0 min. Mixe — Art/Music 
2.7% 14.8 min, 22:4 min. 
4.0% 6.1% 
Science, 


Social/Emotional, 
Social Studies 
12.9 min. 
3.5% 


SOURCE: MDRC calculations based on observations conducted in fall 2015 and completed using the Narrative 
Record (Farran and Bilbrey, 2004). 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Minutes are calculated by multiplying the average proportion of time spent on each type of activity across 
classrooms by the average duration of the full-day classroom observations in fall 2015. 

Spring observations were generally similar to those in the fall. 

aObservers coded an activity content as "None" when it had no instructional purpose, such as meals and 
transitions between activities. 

>Gross motor content develops children's ability to use their bodies through dancing, physical education, games 
at recess, etc. 

‘Observers coded an activity content as "Mix" when no single content area was covered for one continuous 
minute or when multiple activities were occurring simultaneously, such as in center-based play. 
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Table 5.2 


Quality of Math Instruction in Kindergarten, 
Based on Classroom Observations 


Quality Measure Fall Spring 
Percentage of formal math activities in which:* 

The activity was delivered clearly” 97.9 97.9 
The teacher made a mathematical error 4.7 5.4 
Classroom management strategies hindered the activity 16.9 11.3 
Teacher was actively involved in the lesson over 75% of the time® 81.7 86.2 


Percentage of observations in which: 
Teacher changed math materials/content based on individual child skill level - 35.9 


Use of practice (1-5)° 


Teacher asks open-ended questions 2.4 2.0 
Teacher encourages mathematical reflection 2.8 2.7 
Teacher scaffolds children to help them extend their math skills® 7 2.5 
Sample size 
Sites 21 24 
Classrooms 36 39 


SOURCES: MDRC calculations based on full-day and math block observations conducted in fall 2015 and 
spring 2016 using a version of the Classroom Observation of Early Mathematics—Environment and 
Teaching (COEMET; Sarama and Clements, 2007) that was modified for the High 5s study. Math block 
observations were approximately 60 minutes long and were conducted during regularly scheduled math 
instruction time. Full-day observations took place over the entire school day. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

aA formal math activity is defined as one that meets the following criteria: (1) persists for at least 1 
minute, (2) develops mathematics knowledge, and (3) has a discernible topic, goal, and task. This variable 
is calculated for the average classroom by creating the percentage of formal math activities meeting the 
quality benchmark for each classroom and averaging across all classrooms. 

>This item is rated on a 1 to 4 scale (1 = not at all and 4 = almost all of the time), which was transformed 
into a binary variable in which formal math activities with ratings of 3 and 4 were recoded to 1. 

‘This item is rated on a 0 to 4 scale (0 = 0% and 4 = 76%-100%), which was transformed into a binary 
variable in which formal math activities with ratings of 4 were recoded to 1. 

‘This item was not asked in fall 2015; it was added to the instrument for spring 2016 data collection. 

eWhile the wording for each item was specific to that practice, each item (Teacher asks open-ended 
questions; Teacher encourages mathematical reflection; Teacher scaffolds children to help them extend 
their math skills) is rated on a 1 to 5 scale (1 = low or not at all, 5 = high or throughout the activity). This 
calculation takes into account all formal math activities that occurred in the math block observations and 
only those that occurred during the math block portion of the full-day observations. The time-weighted 
average is calculated by multiplying the quality rating by the duration in minutes for each formal math 
activity in a classroom and then averaging across classrooms. 
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degree to which the teacher engaged in mathematical reflection with students and (2) the degree 
to which the teacher asked open-ended questions. Each math activity was evaluated on a scale 
of 1 to 5, where 1 = not at all, 3 = sometimes, and 5 = throughout the activity. These scores 
were weighted by the duration of the math activity, and a weighted average score was calculat- 
ed for every classroom. As shown in Table 5.2, on average, kindergarten classroom teachers 
were rated somewhere between 2.0 and 2.4 on open-ended questions and about 2.8 on reflection 
— indicating that both of these aspects of quality occurred only occasionally. Looking at these 
numbers in a different way, during spring observations only about 8 percent to 13 percent of the 
recorded 172 math activities were given a rating of 4 or 5 on either of these two items. 


There was also relatively little differentiation or scaffolding observed. Differentiated in- 
struction is defined as whether the teacher adapted his or her instruction to suit the skill levels or 
needs of individual children. Scaffolding is similar, but more focused on helping students move 
to the next level of ability. The teacher gives assistance to children based on their current levels 
of ability and provides extra support or additional challenges to help individual children move 
to their next levels of ability. This might be done by providing a different set of materials, or 
just by asking different types of questions when helping a student solve a problem. Teachers 
were observed changing math materials or content for children based on the individual child’s 
skill level in approximately 36 percent of the classrooms observed in the spring. Teachers’ 
average score on a 5-point rating of scaffolding was 2.5. 


In addition, the observed instruction was mostly delivered in a whole-group setting or 
while children were working individually at their seats. As shown in Table 5.3, most (over 80 
percent) of the math instruction involved whole-group instruction or seat work, and about two- 
thirds of the activities either involved no materials (for example, the teacher was talking to the 
children while they were sitting on the rug or at their seats and the children did not have any 
materials) or only a workbook. 


Math Instruction in the Classrooms Compared with High 5s Clubs 


Table 5.4 compares the instruction observed in the kindergarten classrooms with instruction in 
the High 5s clubs, showing more open-ended questions and more mathematical reflection in 
High 5s clubs than in the classrooms. In the spring observations, kindergarten teachers were 
given an average rating of 2.0 on the item asking about open-ended questions, while facilitators 
were rated 2.7 on average. Similarly, teachers were given an average rating of 2.7 on the 
question about mathematical reflection, while facilitators received an average rating of 3.2 on 
this item. Looking at these numbers in a different way, in spring observations 31 percent of 
High 5s facilitators were given a rating of 4 or 5 on the item asking about open-ended questions 
(indicating that the facilitator asked open-ended questions throughout most of the lesson) and 35 
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Table 5.3 


Activity Type and Materials Used During Math Instructional Time, 
Based on Classroom Observations 


Instructional Approach Fall Spring 

Activity type 

Percentage of math instructional time* 
Whole group 64.1 68.8 
Seat work 19.3 16.9 
Small group 4.2 5.0 
Transition 11.7 9.3 
Other” 0.4 0.0 


Materials used by children 
Percentage of formal math activities® 


No materials 37.4 33.1 

Workbooks only 29.4 27.7 

Other materials® 33.2 39.2 
Sample size 

Sites 21 24 

Classrooms 36 39 


SOURCES: MDRC calculations based on full-day and math block observations 
conducted in fall 2015 and spring 2016 using the Narrative Record (Farran and Bilbrey, 
2004) and a version of the Classroom Observation of Early Mathematics—Environment 
and Teaching (COEMET; Sarama and Clements, 2007) that was modified for the High 
5s study. Math block observations were approximately 60 minutes long and were 
conducted during regularly scheduled math instruction time. Full-day observations took 
place over the entire school day. Math instructional time includes math-block-only 
observations and the math block portion of the full-day observations. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

4Percentage of math instructional time is calculated by dividing the total time spent in 
each activity type by the minutes that were included as part of the teacher's regular math 
instruction block for each classroom and then averaging across classrooms. 

bOther represents time that children spent outside, eating meals, and in special 
classes. 

¢Percentage of formal math activities is calculated by dividing the number of formal 
math activities using each material by the total number of formal math activities 
observed in each classroom and then averaging across classrooms. 

dOther includes any materials used other than workbooks, such as math journals, 
manipulatives, or computer. 
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Table 5.4 


Nature of Math Instruction in Kindergarten Classrooms and High 5s Clubs, 
Based on Spring 2016 Observations 


Kindergarten High 5s 
Quality Measure Average* Average 
Teacher asks open-ended questions (1-5) 2.0 2.7 
Teacher encourages mathematical reflection (1-5) 2.7 3.2 
Teacher scaffolds children to help them extend their math skills (1-5) 2.5 2.9 
Teacher changed math materials/content based on individual child 
skill level (% of observations) 35.9 57.7 
Sample size 
Sites 24 23 
Classrooms 39 - 
Clubs - 26 


SOURCES: MDRC calculations based on observations of individual clubs conducted in spring 2016 using an 
instrument developed by MDRC research staff to assess implementation fidelity, and on full-day and math block 
classroom observations conducted in spring 2016 using a version of the Classroom Observation of Early 
Mathematics—Environment and Teaching (COEMET; Sarama and Clements, 2007) that was modified for the 
High 5s study. Math block observations were approximately 60 minutes long and were conducted during 
regularly scheduled math instruction time. Full-day observations took place over the entire school day. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

aThis calculation takes into account all formal math activities that occurred in the math block observations 
and only those that occurred during the math block portion of the full-day observations. The time-weighted 
average is calculated by multiplying the quality rating by the duration in minutes for each formal math activity in a 
classroom and then averaging across classrooms. 


percent of facilitators were rated a 4 or 5 on an item asking about mathematical reflection 
(indicating that the facilitator reviewed key ideas for most or all activities). This compared with 
kindergarten classrooms in which only 8 percent and 14 percent of the math activities observed 
were given a rating of 4 or 5 on open-ended questions and mathematical reflection, respectively. 
Although these items were measured in slightly different ways (ratings were given for each 
activity in the kindergarten classrooms and for the session overall in the High 5s club observa- 
tions), they nonetheless provide some insight into the differences between the two. 


Instructional Content 


Math practices are the specific skills that a teacher tries to address in a lesson (for ex- 
ample, counting forward by Is or adding or subtracting single-digit numbers). During math 
instruction, observers recorded all the math practices that the teacher covered during each math 
activity and identified the primary math practice for each activity. The primary math practice is 
defined as the main practice or topic that the teacher intended to teach. As shown in Table 5.5, 
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Table 5.5 


Percentage Breakdown of Content During Math Instructional Time 
in Kindergarten Classrooms and High 5s Clubs 


Kindergarten Kindergarten —_ High 5s Activities 
Math Activities (%) Fall Observations Spring Observations School Year 
Numbers and operations 94.3 91.3 79.0 
Numeral recognition and writing 8.3 4.4 0.0 
Counting forward by 1s 25.5 21.2 0.0 
Recognizing quantity without counting 1.3 0.0 3.1 
Complex counting 1.3 6.6 24.6 
Comparing and ordering 24.2 6.6 11.3 
Composing numbers 28.0 13.9 22.6 
Adding and subtracting 5.7 38.7 17.4 
Other math content areas 5.7 8.8 21.0 
Patterning 1.9 0.0 4.6 
Shapes 2.6 7.3 11.8 
Measurement 0.0 0.0 4.6 
Other 1.3 1.5 - 
Sample size 
Sites 21 24 - 
Classrooms 36 39 - 
Math activities* 157 137 195 


SOURCES: MDRC calculations based on the High 5s Full-Scale Curriculum developed by the University of 
Michigan and on full-day and math block observations conducted in fall 2015 and spring 2016 using a version 
of the Classroom Observation of Early Mathematics—Environment and Teaching (COEMET; Sarama and 
Clements, 2007) that was modified for the High 5s study. Math block observations were approximately 60 
minutes long and were conducted during regularly scheduled math instruction time. Full-day observations took 


place over the entire school day. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Math content categories are based on the primary math practice categories from the COEMET. Complex 
counting includes counting backward by 1s, counting forward or backward by 10s, skip counting by a number 


other than 10, and counting forward by 10s and 1s. 


For kindergarten classrooms, data on the math content of activities is based on observations at two time 
points during the school year; coding of the GoMath! curriculum, which was used in most classrooms, 
indicates that the activities implemented during these two time points are representative of the entire 
curriculum. For High 5s, coding of math content was based on all activities in the curriculum; fidelity data 
indicate that the planned activities in the curriculum were closely aligned with the activities that were actually 


implemented. 


ln kindergarten classrooms, a math activity is defined as a formal math activity observed using the 
COEMET, which must meet the following criteria: (1) persists for at least 1 minute, (2) develops mathematics 
knowledge, and (3) has a discernible topic, goal, and task. All formal math activities observed in the math 
block observations are included. In the full-day observations, only formal math activities taking place during 


regularly scheduled math instruction time are included. In High 5s, a math activity is defined as either a start- 
up or main activity as outlined in the High 5s Full-Scale Curriculum. 
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the primary math practices observed in the fall were most commonly composing numbers — a 
precursor to addition and subtraction in which students become fluent with all the ways to 
“make” a number (for example, 4 is 1 and 3, 2 and 2, 4 and 0, and so on) — counting forward 
by ls, and comparing and ordering (for example, knowing that 9 is greater than 5). As expected, 
instruction moved from these basic math practices to more sophisticated ones later in the year. 
In the spring, significantly more addition and subtraction and less comparing and ordering and 
composing numbers were observed. Relatively few math practices that were not related to 
numbers and operations, such as geometry, measurement, or patterning, were observed. Chap- 
ters on geometry and measurement come at the end of the district-supported Go Math! curricu- 
lum, and it is possible that teachers may have covered these topics later in the year. However, as 
is often the case, preliminary analyses of pacing in these classrooms suggest that most teachers 
would not have reached these chapters by the end of the year. 


As Table 5.5 indicates, High 5s exposed children to a wider range of content. While 
fewer than 10 percent of classroom activities focused on geometry, measurement, and pattern- 
ing, over 20 percent of High 5s activities focused on these content areas. In both the fall and the 
spring, High 5s clubs included fewer activities focused on basic math skills (for example, 
counting forward by 1s and numeral recognition and writing). 


Summary 


Overall, these findings suggest that the instruction in the High 5s clubs differed in a number of 
ways from the regular kindergarten instruction children received. First, the High 5s program 
presented a wider range and somewhat more advanced instructional content than was observed 
in kindergarten classrooms. Classroom teachers were, for the most part, following the district’s 
Common Core-aligned kindergarten math curriculum, which focused on numbers and opera- 
tions for a majority of the year. Geometry and measurement are introduced near the end of the 
Go Math! curriculum, while geometry, measurement, and patterning activities were included 
throughout the year in High 5s. 


Second, content was, by design, delivered in a small-group format in the High 5s clubs, 
and all the High 5s activities involved the use of hands-on manipulatives. By comparison, there 
was very limited small-group instruction in the kindergarten classrooms, and about two-thirds 
of the activities observed in kindergarten classrooms involved only workbooks or no materials 
at all. One reason for the difference may be that in many classrooms there was only one teacher 
present, and it is difficult for kindergartners to participate in small-group activities without adult 
supervision. 


Third, facilitators asked more open-ended questions and engaged in more differentia- 
tion of material than was observed in kindergarten classrooms. Differentiation and mathemati- 
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cal reflection were difficult for the High 5s facilitators, who were addressing the needs of just 
four students at a time, all of whom had been exposed to the Building Blocks curriculum in pre- 
K. Differentiating instruction and encouraging mathematical reflection is even more challenging 
in a classroom of 20 children with substantial variation in experience. 


Finally, as discussed in Chapter 4, High 5s students received an additional 75 minutes 
of math instruction per week above what they would have received in their regular classrooms 
(approximately 280 minutes per week). This is an increase of more than 25 percent in mathe- 
matics instructional time each week. As will be discussed in more detail in the final chapter, 
these factors may have contributed to the program’s positive impacts on achievement. 
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Chapter 6 


Implications of the Findings 


The High 5s curriculum was designed to pick up where the Building Blocks curriculum left off 
in prekindergarten and to move children along four key mathematical trajectories by engaging 
them in mathematical games that are fun and developmentally appropriate. High 5s was a test of 
whether it was possible to create and deliver aligned math enrichment across pre-K and kinder- 
garten and whether that alignment had the potential to sustain the impacts of a high-quality 
math intervention in pre-K. 


As described in detail in this report, the implementation of the High 5s program 
achieved the goals that were initially set out for the program — clubs met regularly, the activi- 
ties were delivered consistently and accurately, and attendance and engagement were high 
throughout the year. Substantial resources (from both MDRC and Bank Street College of 
Education) were expended to ensure the high level of implementation. Staff members from 
MDRC and Bank Street worked closely with families to ensure high rates of attendance and 
cooperated with schools to ensure that there was time and an appropriate space for the clubs to 
meet. The Bank Street supervisors and program director worked closely with facilitators 
throughout the year to help ensure the curriculum was delivered effectively, providing a level of 
supervision and professional development beyond what most early career teachers receive. 


As noted in a companion report, the program had a positive and statistically significant 
impact on the math skills of the children who participated in the High 5s program.! The pro- 
gram produced positive impacts (effect size = 0.19 standard deviations) on one of two measures 
of mathematical knowledge and skill when students who received High 5s in kindergarten and 
Making Pre-K Count in pre-K were compared with students who received only Making Pre-K 
Count in pre-K. The impacts were larger when students who received High 5s were compared 
with a group of children who received no math enhancement in pre-K or kindergarten, which 
suggests that the two years of enhanced math instruction had an additive effect. 


Potential Contributions of the High 5s Program 


The findings in this report suggest a number of potential mechanisms through which High 5s 
may have contributed to the development of children’s math skills. First, High 5s led to addi- 
tional mathematics instructional time: Students in High 5s received, on average, 75 minutes of 


'Mattera, Jacob, and Morris (2018). 
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math instruction each week over and above the approximately 280 minutes they were receiving 
in class. This is an increase of over 25 percent. 


Second, High 5s had a different instructional approach from kindergarten classroom in- 
struction. In classrooms, 83 percent of math time was spent in either whole-group instruction or 
seat work, and most activities involved either workbooks or no materials at all. High 5s instruc- 
tion occurred in small groups or individually with the facilitator (on Game Days, some facilita- 
tors worked one on one with individual students). High 5s was primarily game-based and 
provided students with opportunities to engage with mathematical concepts using a variety of 
manipulatives and materials. These instructional practices were highly aligned with the Building 
Blocks curriculum, and this consistency may have contributed to student learning in High 5s. 


High 5s also exposed students to more advanced mathematical topics. As shown in 
Chapter 5, High 5s clubs covered a wider range of content than did kindergarten classrooms and 
more time was spent on more advanced mathematical skills. 


Finally, the instructional climate in clubs may have differed from the climate in class- 
rooms. The following exchange, as reported by one of the facilitators during the last week of the 
clubs, captures this difference: 


While filling out their “why we like math” page, the children [in this club] all 
concluded that they didn’t like math. I [the facilitator] said that was strange be- 
cause they’ve been doing math in High 5s all year and were loving it and were 
so happy. They clarified that they like math in High 5s but they don’t like it in 
school. R explained in his words that “In school, you do math and you be quiet 
and look down at your paper. They just tell you that you’re wrong. And then no- 
body talks to you. It’s just wrong and you have to be quiet. But in High 5s we 
have you. You never say we did it wrong and we all talk and figure it out and 
then nobody’s wrong. That’s why I’m happy when I do math in High 5s.” 


Another facilitator wrote at the end of the year that the most rewarding aspect of the job 
was “When the kids finally ‘get’ a new concept and they become more confident in themselves 
it honestly makes it all worthwhile. One of my students wrote in her high 5s book that one of 
the things she learned in high 5s was to ‘never stop trying.” The extensive training and ongoing 
supervision that facilitators received from Bank Street likely contributed to this instructional 
climate. 


There is some evidence to support each of these potential contributions, and any effects 
of the program likely arise from some combination of all four. It is therefore important to be 
cautious in concluding that just one or two of these elements would be sufficient to produce 
impacts. 
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Program Replicability 


Given the program’s impact on math skills, it is worth considering whether High 5s could be 
replicated in other contexts. There were many factors that led to the approach to implementa- 
tion described in this study, including the requirements of the research design itself. However, 
it is likely that the program could be operated more efficiently and with fewer resources if 
implementation were approached in a different way. For example, in this study facilitators 
were required to travel from school to school, which expended both time and money. Similar- 
ly, holding the clubs outside of regular school time limited the times each day in which clubs 
could be held, required working closely with schools to find an appropriate time and place for 
the clubs, and required considerable coordination with families to help ensure high rates of 
attendance. 


Many of these factors could be avoided if the program were run by staff members al- 
ready in the schools and if clubs were held during the regular school day. Using paraprofession- 
als already within the school building would offer greater flexibility in scheduling and would 
eliminate commuting between schools, which was challenging for many of the facilitators. Such 
a model might have other benefits as well. Paraprofessionals in the school might have more 
experience working with small groups of children and might be better equipped to mitigate the 
behavioral challenges that the less experienced facilitators in this study faced. Pulling students 
out during the school day would also reduce attendance problems, might reduce the challenge of 
finding appropriate space, and could potentially be more conducive to instruction because 
students would be less tired. 


Table 6.1 describes the set of resources that would probably be required to replicate the 
program with school-based staff members during the regular school day. Assuming that schools 
would want to serve most if not all of their kindergarten students, the numbers in the table 
reflect the following assumptions: Each school has three classrooms and 20 children per 
kindergarten classroom, for a total of 60 students. Students would be divided into 15 clubs with 
four children per club. Each of these 15 clubs would meet three times a week. To fit this many 
clubs into a typical school week would require that some clubs meet concurrently. Staffing 
could include three half-time facilitators each running 5 clubs, or one full-time facilitator 
running 10 clubs and one half-time facilitator running 5. Assuming that each session lasts 
around 45 minutes (including setup and transitions between clubs), running 5 clubs would 
require approximately 10 to 15 hours of staff time per week, plus an additional 5 hours weekly 
for preparation, supervision, and training. 


As reflected in the table, in addition to the facilitators’ time, schools would need to take 
into account the costs of materials, supervision, training, and administrative support. Schools 
would also need to identify appropriate spaces to hold the clubs. The assumptions made above, 
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Table 6.1 


Potential Resources Needed to Replicate the High 5s Program During 
the School Day Using School-Based Staff 


Resources 


Amount 


Notes 


Facilitators 


Supervisor 


Program director 


Administrative support 


Materials: High 5s kits 
and curriculum materials°® 


Training 


1-1.5 FTEs 


0.5 FTE 


0.10 FTE 


0.05 FTE 


1 per facilitator 


1 week of staff time 
before clubs begin 


Estimate assumes clubs each last about 45 minutes, 
including setup and transition time. 

To run 5 clubs meeting 3 times a week would require about 
10-15 hours of staff time, plus an additional 5 hours for 
preparation, supervision, and training. 

Staffing could include 3 half-time facilitators who each run 5 
clubs or 1 full-time facilitator running 10 clubs and 1 half- 
time facilitator running 5 clubs.? 


In the High 5s implementation described in this report, each 
supervisor was responsible for a cohort of 5 facilitators. One 
supervisor could be shared between 2 schools. 


In the High 5s implementation described in this report, a 
program director provided support to the supervisors and 
helped ensure that the program was implemented consis- 
tently. 

If the program were being implemented across an entire 
district, the costs of a program director could be shared 
across all schools.° 


The program requires some administrative support to run 
effectively: help ordering and distributing supplies, coordinat- 
ing scheduling, identifying appropriate spaces to hold the 
clubs, etc. 


The cost for materials is around $25 per child, depending on 
the number of children served. 


This might require overtime or summer pay or release time 
and potential food or space. 

In the High 5s implementation described in this report, there 
were also additional training days held throughout the year 
on days school was not in session. 


SOURCE: MDRC calculations based on resources required to implement the program. 


NOTES: FTE = full-time employee. 
The table assumes 20 children per kindergarten classroom and three classrooms per school for a total of 60 

students. Students would be divided into 15 clubs with four children per club. Each of these 15 clubs would meet 

three times a week. To fit this many clubs into a typical school week, some clubs would have to meet concurrently. 
alf a school system decided to share facilitators across multiple schools, time and transportation costs would need 


to be included. 


>If the program were adopted by a single school (as opposed to an entire district), a program director might not be 


needed. 


°Materials include items such as the curriculum binder, manipulatives, activity-specific materials, and supplies. 
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on which the table is based, could change depending on how the program was implemented. For 
example, in a large school where the High 5s meeting room was a long way from kindergarten 
classrooms, more than 45 minutes might be required to move students to and from clubs and 
run a 30-minute club session. Or if the program were run as a push-in rather than a pull-out 
program, perhaps less time would be required and a designated space would not be needed, but 
it might be difficult to keep students focused in a classroom where other activities were also 
occurring. If the program were adopted by a single school (as opposed to an entire district), a 
program director might not be needed. If a school system decided to share facilitators across 
multiple schools, time and transportation costs would need to be included. High 5s could also be 
implemented as part of an after-school program, but it would probably require a larger number 
of facilitators if the goal were to serve all or most of the participating kindergarten students, 
since there would be a limited period of time in which to hold the clubs. In short, the program 
could potentially be structured in a variety of ways, and each approach would require a some- 
what different set of resources. 


A number of components of the current implementation that are harder to quantify may 
also have made a difference. First, as noted earlier, three of the five program supervisors and the 
program director involved in the implementation of High 5s had previously been a part of the 
Making Pre-K Count study; thus they already had a strong background in the learning trajecto- 
ries approach and the pedagogical aspects of the Building Blocks program. This may have 
contributed to the strong implementation. Similarly, the project benefited from the infrastructure 
and experience of Bank Street College of Education, which had both experience and expertise 
in early childhood education as well as experience working in and credibility with New York 
City schools. The project was also able to tap into several colleges and universities, as well as a 
large network of education professionals in New York City, to recruit a strong and committed 
set of facilitators. 


It might be difficult to replicate some of the instructional aspects of the High 5s model. 
Facilitators received a substantial amount of training and supervision both before clubs began 
and throughout the school year — much more support than is typically provided to a first-year 
teacher. In addition to the week of training shown in Table 6.1, facilitators recetved ongoing, 
daylong trainings throughout the year on days when schools were not in session. School staff 
members might not have the flexibility to attend this amount of training and schools might not 
have the resources to provide it. 


The training and supervision infrastructure in the current implementation model was 
adopted to help ensure that the program could be implemented with fidelity right from the 
start, since program impacts were going to be measured at the end of the first year of imple- 
mentation. It also helped ensure that the learning environment in clubs remained positive. 
Without a consistent message regarding the philosophy of instruction, clubs might have taken 
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on the instructional climates of their schools, which varied widely. Although the curriculum is 
semiscripted and self-explanatory in its approach, ensuring that it is delivered with the highest 
level of quality may require more supervision and training than is reflected in Table 6.1. 


Finally, the curriculum itself was designed to align with and build on what children in 
the study had experienced in pre-K. The study was not designed to demonstrate how effective 
the curriculum would be for children who did not have this grounding in preschool. Further 
research is needed on how these activities and instructional setup would work for children who 
may have had a different set of experiences in the year before kindergarten. 


Conclusions 


Despite uncertainty in the planning phases about the viability of a math club model like High 
5s, the program was implemented well, with high attendance rates and student engagement. 
Staff members hired at a paraprofessional level, many with limited or no formal teaching 
experience, were, with the support of a strong training and supervision model, able to provide a 
wide range of math content with fidelity to the program model. The program had a robust, 
statistically significant impact on one of two measures of math achievement. A number of 
factors may have contributed to the positive impacts on math, including additional time on 
math, differences in content covered between classrooms and math clubs, and differences in 
both instructional approaches and instructional climate. 


There is an extensive body of research on the effectiveness of reading instruction for 
young children in small groups? and of math tutoring for older students.? However, at the time 
the High 5s program was implemented, there was much less evidence on the effectiveness of 
small-group tutoring for boosting math achievement in the early grades. This study adds to the 
growing body of evidence that suggests that small-group instruction, even when delivered by 
paraprofessional-level staff, can be an effective way to improve math skills. 


There are two potential ways in which small-group enrichment may have contributed to 
the positive outcomes observed in the study. First, small-group work could be one way to 
provide kindergarten instruction that is more closely aligned with the pre-K experience. Pre-K 
tends to involve a substantial amount of small-group instruction with many hands-on learning 
opportunities, and the High 5s program continued this approach. Instruction that is more closely 
aligned with the pre-K experience could help mitigate the fade-out that is often associated with 
high-quality pre-K experiences. At the same time, in the model tested in this study, not only the 
mode of instruction but also the content covered was aligned with children’s pre-K experiences, 


Slavin, Lake, Davis, and Madden (2010). 
3Fryer (2011); Smith et al. (2013). 
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and it is unclear whether a program like High 5s could work in the absence of two years of 
enhanced and aligned instructional content. 


In addition, the small-group experiences that a math club model offers might help en- 
sure that children get more individualized instruction and do not “fall through the cracks.” 
Individualized instruction may be difficult to achieve in a classroom with only one teacher and 
many children with varying needs. In a small-group setting, it is easier for teachers to assess 
each child’s level of understanding, ensure that the children are engaging in assigned tasks, and 
differentiate instruction. 


The positive impacts of the High 5s program demonstrated in this study suggest that it 
would be worth examining whether the program could work in different contexts. For example, 
would the model be effective in scenarios in which children come into kindergarten with a 
variety of different pre-K math experiences? Would the program be effective if schools use in- 
house staff members to deliver the clubs, or staff members without the level of training and 
supervision that facilitators were provided in the model that was tested? It is important to 
understand how the different components of the program work and for whom. 


It is also worth considering whether the appropriate next step is to try to integrate more 
small-group work and hands-on learning opportunities into kindergarten math instruction in the 
classroom. This was challenging in the schools in this study, where there was often only one 
adult in the classroom. A pull-out or push-in math club program like High 5s that operates 
during the regular school day could provide such an opportunity. Additional research is needed 
to investigate whether the program could be successfully implemented in these ways or whether 
there are other ways to integrate small-group work and hands-on activities into the regular 
school day. 
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Appendix A 


High 5s Sample Lesson 


Chocolate Chip Count 


Early Addition and 


Subtraction 


Students will subitize (know amount without counting) the number of dots on two 
separate cards and then use a method to find the sum. To find the sum, they may 


need to count the objects or use their fingers. 


You will need: 


Activity Description: 


5) 


O Printed monster with 


in mouth, placed over a 


container 


cards 


Oo 5 sets of counting cards (0- 


O 4 sets of numeral cards 


oO Bowl or small trash can 
O 12 laminated blank numeral 


slit cut 


In this game, students imagine that the cards 
are brownies with chocolate chips and that a 
monster is eating the brownies. After the 
students have determined the number of 
chocolate chips on two brownies, they write 
that number on a blank numeral card and 
‘feed’ it to the monster. At the end of the 
game, the students can look inside and see 
what the monster ate. 


Student models addition 
problems by counting 
each set and joining them, 
then counting from 1 to 
find the total. 


Early Addition and Subtraction 


Student models addition 
problems as previously and 
also models subtraction 
problems by getting the larger 
set, counting to remove the 
smaller set, then counting to 
determine what remains. 
Student may count from one to 
add two sets 


Student can count on to add sums 
in the range of 20, which includes 
counting from a number other 
than 1 and tracking the number 
of counts necessary (e.g. to do 7 
+ 4 the student may count “8, 9, 
10, 11,” while using fingers to 
keep track of the counts). 


Activity 
Set-Up: 
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Concept: 


Building early 


strategies 


Today, the students will practice addition by determining the number of chocolate chips that 

are eaten by the monster. The students will use counting cards to represent brownies, and write 
numbers on blank numeral cards to represent the sums that will be fed to the monster. Lay out the 
stacks of counting cards and create sets of numeral cards that correspond to the students’ 
learning trajectory (above). 


Place the bowl or small trash can on the table, and tape the monster to it so that the container 
can catch the brownies as the students feed the monster. 


Introduce the game to the students. 


* “Do you like chocolate chips on brownies?! Do you think this monster can eat a lot of 
chocolate chips? Today you get to tell me how many chocolate chips are on the brownies. 
Then, we feed them to the monster!’’ 


Now, you can model one example for the students. 


*‘I’m going to show you a brownie and | want you to see how many chips are on it. I'll turn 
over a card for just a moment, and then hide it again, so watch closely?’ 


Take two counting cards that are appropriate for the students’ learning trajectory. Place the 
cards face down on the table. Flip the first card so that the students can see the dots for about 
2 seconds, then turn it face down again. Say: 


* “How many chocolate chips did you see on the brownie? (Allow the students to answer.) 
Remember that number, because we’re going to feed the monster another brownie!” 


> If the students are new to addition, you may want to use two very small numbers for this 
example. 


Turn over the second card so that students can see how many chocolate chips are on the second 
brownie. 


® “How many chocolate chips did you see on that brownie? (Allow the students to answer.) 
* “Now, how many chocolate chips altogether?” (Give the students time to answer.) Tell 


the students that now they get to feed the monster! Write the sum on a blank numeral 

card. For example, if the monster ate a brownie with 2 dots and a brownie with 1 dot, write a 
3 on the numeral card. Have a student feed it to the monster by placing the card through the 
monster’s mouth and into the container. Now that you’ve done one example, you’re ready to 
play the game! 


Begin play again. This time, have another student write the sum on the numeral card that 
is needed to feed the monster. Here are some strategies and tips for helping students 


play. 


When you ask students to add the number of chips on the two brownies, it is appropriate 
for them to do the problem without looking, although some students may count on their 
fingers. If they hesitate or have trouble adding, ask: 


*“Do you remember how many chocolate chips were on the first brownie?” (Listen for 
response.) 


®\f the response is correct, acknowledge the students’ success and ask, “How many on the 
second brownie?” (Listen for response.) 


®If the students give an incorrect response about the first or second brownie, ask, “‘Do 
you need fo see that again?” (Turn over the brownie and leave it right side up for 
the students to see if needed.) 


*Once the students have identified the correct number of chocolate chips on each 
brownie, ask: “And how many chips are there altogether?” 


> If the students need to look at the cards, flash the first card, then show them the 
second card to see if they can first think of the number from the first brownie. Then, 
have the students count on the number of dots on the second brownie. If the students 
are not successful, they can look at both cards. The goal is for the students to add 
with as little support as possible, but to provide support if needed so that they don’t 
get frustrated. 


When students have successfully identified a sum, they get to feed the monster. You can 
say: 


*“‘Now we get to feed the monster! Let’s write the number and feed it to him.”’ (Have 
students write the number on a blank card.) Look how many chocolate chips he is 
eating!’ 


Students may use a variety of strategies to determine the sum of the chocolate chips. Here 
are some examples of how students might solve 5 + 3. 


Students who are counting all of the sets (red) will need more practice with small amounts 
of dots on the brownies. They may count the first set, count the second set, then count both 
sets: “1, 2,3, 4,5, ... 1,2, 3 ....1, 2, 3, 4, 5, 6, 7, 8.” 


Students who can work with slightly larger numbers (orange) may be able to use a 
numeral card and then continue counting the dots on a brownie. They may also be able to 
work in pairs. Students at this point in the trajectory may count the first set then continue 
counting on through the second set: “1, 2, 3, 4,5... 6, 7, 8.” 


Students who can count on can use two numeral cards to represent the chocolate chips on 
the brownies and can work in pairs. They can count on from the number of the first set: 
"D200 Fy Bo 
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To support students in focusing on the addition, you could ask questions like this: 


*‘‘How many chips on the first brownie? And how many chips on the second brownie? How 
many did you say that was altogether?” 


*“Let’s say that with addition! (First number) plus (second number) makes (sum).”” 


> Encourage playfulness by reminding the students that the monster loves chocolate 
chips, just like they do! You could also tease them by pretending to be worried about 
how many chocolate chips the monster is eating. 


Play the game until the monster is completely “full” of chocolate chips, or until the cards are 
gone. When the students have played for a while and fed the monster each time, you can 
invite them to peek and see how many chocolate chips the monster ate. 


*®“Let’s look at all the chocolate chips the monster ate today.” 


*“‘Want to peek? Oh my! He ate a lot of chocolate chips! My tummy would be upset if | ate 
that many chocolate chips!” 


It’s not important to come up with this total. Students may just want to see all that the 
monster has eaten. 


This activity is most conducive to group work rather than pairs. Once the students are 
comfortable with the game and you are feeding the monster for the second or third time, 
you may want to remind them of how to play. 


Breaking It Down 


First you do this: Next you do this: Then you do this: 
Look at two brownies Figure out how many chocolate Write the number on a card 
(counting cards). chips are on the brownies. and feed the chocolate chips 
to the monster! 
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Feedback Phrases: You could say... 
®“‘How many chocolate ®“‘How many are on the first ®“‘How many are on the first 

chips are on this first brownie? How many are brownie? How many are on the 
brownie? How many on the second brownie?” second brownie?” 
are on the second Flip both cards down Flip both cards down after the 
brownie?” after the students have students have seen them. 

seen them. 

*“Let’s count them ® “How many is that altogether?” 

altogether.” ® “How many is that Students at this level may find 

altogether?” success counting on. You may 
To encourage Students in this level may want to encourage them to 
counting on: make use of their fingers work with sums that are close 

®“‘How many are on this and could count from one to ten. 

brownie? (Flip the or begin by counting on. 
card so the dots can’t *“‘How many is 10 chocolate 
be seen.) To encourage counting on, chips and 6 chocolate chips?” 
Lef’s keep counting you could increase the 
on with this number of dots on the *“So how many is 9 chocolate 
brownie.” (Count first brownie: chips and 6 chocolate chips?” 
on the dots on the ®“‘How many are on this 
second brownie.) brownie? (Flip the card so 

the dots can’t be seen.) 

Let’s keep counting on with 

this brownie.” (Count on 

the dots on the second 

brownie.) 


® “J noticed you doing addition today. When you say how many chips there are altogether 
on the two brownies, you are adding. Sometimes you did this by counting and other 
times you just knew!” 


® “Think for a moment and hold your answer while | ask you one more question. Tell me 
how many chocolate chips | would have if my first brownie had 4 chips and my second 
brownie had 3 chips?” (Wait for all students to think about this problem, then have 
one student share.) 


® “The more you do addition like this, the better you will get at it.” 
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Appendix B 


Curriculum Pacing Guide and Sample Activities 


6S 


Curriculum Pacing Guide 


ae Main Activity 
ena faint 15-20 minutes Family Note 
7-10 minutes ; 
(Trajectory) 
Getting to Know You High 5s Expectations 
Blast Off Guess My Number | Build Stairs Counting 
Blast Off Guess My Number | Shake and Compare Composition of Number 
: Family Note: 

Blast Off Guess My Number | Pattern Repeater Patterning 


Shake and Compare 


Game Day 


ae Main Activity 
Start-Up Activit i 
acngetes 15-20 minutes Family Note 
7-10 minutes ; 
(Trajectory) 
Number Mat Dump and Sing Longest Train Counting 
Number Mat Dump and Sing Chocolate Chip Count Early Addition and Subtraction 
: F : Family Note: 

Number Mat Dump and Sing Clothesline Numbers Counting 


Clothesline Numbers 


Game Day 


09 


ae Main Activity 
vere aaaly 15-20 minutes Family Note 
7-10 minutes ; 
(Trajectory) 
Ollie Otter Break and Make | Shake and Compare Composition of Number 
Ollie Otter Break and Make | Chocolate Chip Count Farly Addition and Subtraction 
Ollie Ot Break and Make | Longest Train Counti Early Note: 
ie Otter reak a ake ongest Train Counting Chocolate Chip Count 
Game Day 
roie Main Activity 
sealed ob 15-20 minutes Family Note 
7-10 minutes : 
(Trajectory) 
Number Mat Guess My Number | Clothesline Numbers Counting 
Number Mat Guess My Number | Chip Stack Farly Addition and Subtraction 
Number Mat Gusts My Noriber Building With Shapes — Puzzle Maker — Day 1 Family Note: 
Geometry Building Shapes 1 
Game Day 


19 


ete Main Activity 
Pea polity 15-20 minutes Family Note 
7-10 minutes : 
(Trajectory) 
Number Betore: | ‘shape Ser Chocolate Chip Count Early Addition and Subtracti 
and After ape Sor ocolate Chip Cou arly ition and Subtraction 
Number Before : , 
and Atle: Shape Sort Clothesline Numbers Counting 
Number Before : 2 : Family Note: Num- 
and After Shape Sort Chip Stack Early Addition and Subtraction ber Before and After 


Game Day 


af Main Activity 
Start-Up Activit 0 
Sladen! 15-20 minutes Family Note 
7-10 minutes ; 
(Trajectory) 
Ollie Otter Dump and Siig Building With Shapes - Puzzle Shapes — Day 2 
Geometry 
Ollie Otter Dump and Sing Fish for a Number Composition of Numbers 
3 * Pa . ai : Family Note: 

Ollie Otter Dump and Sing Addition Bingo Early Addition and Subtraction Ollie Otter 


Game Day 


co 


are Main Activity 
ste poy, 15-20 minutes Family Note 
7-10 minutes " 
(Trajectory) 
Dot Strips Break and Make _ | Chip Stack Early Addition and Subtraction 
Dot Strips Break and Make | Fish fora Number Composition of Number 
Dot Strips Break and Make Building With Shapes — Shape Maker - Day 3 Family Note: 
Geometry Dot Strips 
Game Day 


nite Main Activity 
alle nelly 15-20 minutes Family Note 
7-10 minutes ; 
(Trajectory) 
Number Before ses 3 oe . 
Sal Mba Shape Sort Addition Bingo Early Addition and Subtraction 
Number Before ; ia 
sind Ata Shape Sort Fish for a Number Composition of Number 
Number Before ‘ tt : Family Note: 
and Afar Shape Sort Sammy the Squirrel Early Addition and Subtraction Shape Sort 
Game Day 


Start-Up Activity 


Main Activity 


é 15-20 minutes Family Note 
7-10 minutes . 
(Trajectory) 
Number Elp = Find a Partner Addition Bingo Early Addition and Subtraction 
Decade 
Number Elipi Find a Partner Memory Composition of Number 
Decade 
Number Flip - . : es ; Family Note: 
Decade Find a Partner Sammy the Squirrel Early Addition and Subtraction Find a Partner 
Game Day 
- a6 Main Activity 
et) Start-Up Activity Tonite: Family Note 
7-10 minutes f 
(Trajectory) 
Dot Strips ecg How Much Bigger? Early Addition and Subtraction 
Dot Strips rae Comparison Bingo Farly Addition and Subtraction 
é What's the as Family Note: 
Dot Strips Number? Memory Composition of Number What's the Number? 


Game Day 


v9 


Start-Up Activity 


Main Activity 
15-20 minutes 


Family Note 


7-10 minutes ; 
(Trajectory) 

Core Unit ‘i os F 

Go Round Sammy the Squirrel Early Addition and Subtraction 
Concert 
a Go Round How Much Bigger? Early Addition and Subtraction 
Concert 
Core Unit ‘ : de ; Family Note: 
Concert Go Round Comparison Bingo Early Addition and Subtraction Core Unit Concert 


Game Day 


Start-Up Activity 
7-10 minutes 


Main Activity 
15-20 minutes 
(Trajectory) 


Family Note 


Busy Beaver 10s _ | What’s the ; 

sate Number? Pattern Repeater 2 Patterning 

Busy Beaver 10s _ | What’s the a 

Sede Number? Memory Composition of Number 

Busy Beaver 10s _ | What’s the . * - : Family Note: 

or ls Number? How Much Bigger? Early Addition and Subtraction How Much Bigger? 


Game Day 
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Start-Up Activity 


Main Activity 


: 15-20 minutes Family Note 
7-10 minutes . 
(Trajectory) 
Number Flip - Core Unit : : a . 
Decade Concert Comparison Bingo Early Addition and Subtraction 
Number Flip - Core Unit : : 
Bacade Concert Chilly Cherry Counting 
Number Flip - Core Unit : : me : Family Note: 
Decade Concert Magic Number Trick Early Addition and Subtraction Medic Number trick 
Game Day 
a6 Main Activity 
Start-Up Activit F 
ae! 15-20 minutes Family Note 
7-10 minutes i 
(Trajectory) 
sear Beaver I0s Go Round Measurement Olympics 1 Measurement 
Busy Beaver 10s ‘ * yi ‘ 
oris Go Round Magic Number Trick Early Addition and Subtraction 
Busy Beaver 10s Go Round Chilly Cherry Counting Pari Ney: 


orls 


Busy Beaver 10s or Is 


Game Day 


Note: Consult your supervisor on the timing of this Personalized Family Note. It should be sent home prior to the school year’s Spring 


Break week. 
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Main Activity 


alt pay 15-20 minutes Family Note 
7-10 minutes : 
(Trajectory) 
F Finn Prog dump Paper Folding Cat Geomet 
armer Finnegan | ia cure aper Folding Cat Geometry 
Farmer Finnegan Frog cumip Chocolate Chip Monster Early Addition and Subtraction 
Measure 
F Finn Frog Jump Magic Number Trick Early Additi d Subtracti Family Note: 
armer Finnegan | aoe ore agic Number Trick Early ition and Subtraction Chocolate Chip Monster 
Game Day 
mere Main Activity 
Start-Up Activit ° 
Dae 15-20 minutes Family Note 
7-10 minutes Z 
(Trajectory) 
Is it or Not? Find a Partner Chilly Cherry Counting 
Is it or Not? Find a Partner Paper Folding Boat Geometry 
: : ; Family Note: 
Is it or Not? Find a Partner Crack the Code Counting A 
Paper Folding 


Game Day 


Main Activity 


Start-Up Activit : 
oe » 15-20 minutes Family Note 
7-10 minutes . 
(Trajectory) 
Busy Beaver 10s : Chocolate Chip Monster Early Addition and 
Special Squares : 

and ls Subtraction 
Busy Beaver 10s . : 
and 1s Is it or Not? Measurement Olympics 2 Measurement 
Busy Beaver 10s Family Note: 


Is it or Not? 


Phil the Baker Early Addition and Subtraction 


L9 


Note: 


and Is Is It or Not? 
Game Day 
ae Main Activity 
Start-Up Activit F 
Ace 15-20 minutes Family Note 
7-10 minutes a‘ 
(Trajectory) 
- Frog Jump Chocolate Chip Monster Early Addition and 
Farmer Finnegan : 
Measure Subtraction 
Farmer Finnegan prog Cumip Crack the Code Counting 
Measure 
p Frog Jump ‘ Family Note: 
Farmer Finnegan Measure Shape Mosaic A Geometry Crack the Code 


Game Day 


Consult your supervisor on the timing of this Assessment Day. 
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Start-Up Activity 


Main Activity 


7-10 minutes 15-20 minutes Family Note 
(Trajectory) 
iia nae Phil the Baker Early Addition and Subtraction 
Chip Flip Make the j 
Century Number Pattern Repeater 2 Patterning 
Chip Flip Make the : oe : Family Note: 
Century Number Cherry and Mr. S’more Early Addition and Subtraction Chemy and Mr: $'riiore 


Game Day 


Note: Pacing Guide Day 84 Wrap-Up Activity preparation: Facilitator/student photos should be taken to allow enough time for printing. 


ae Main Activity 
Start-Up Activit : 
We ul 15-20 minutes Family Note 
7-10 minutes ‘ 
(Trajectory) 

re a “ Special Squares | Measurement Olympics 3 Measurement 
Busy Beaver 10s ‘ eo ; a F 
anal Tie Is it or Not? Cherry and Mr. S’more Early Addition and Subtraction 
Busy Beaver 10s Family Note: 


Is it or Not? 


and Is 


Crack the Code Counting 


Measurement Olympics 


Game Day 
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Ac Main Activity 
ara Paueity 15-20 minutes Family Note 
7-10 minutes Z 
(Trajectory) 
Chip Fup Matte'the Phil the Baker Early Addition and Subtraction 
Century Number 
Chip Flip Make the ; 
Century Number Shape Mosaic B Geometry 
Chip Flip Make the ; ir : Family Note: 
Century Numbar Cherry and Mr. S’more Early Addition and Subtraction Phil the Baker 
Make Ten High 5s Bump Early Addition and Subtraction 
Get the Carrot Crack the Code Puzzles Counting 
How Much Farther? Building with Shapes #3 Geometry 
Wrap-up Activity (Wrap-Up Day 1) 
Open Choice/Play Favorite Game(s) (Wrap-Up Day 2) 


Note on Wrap-up/Final Day: It is possible not all clubs will have time for 83 meetings, plus the Wrap-up and Final Day activities. Consult your 
supervisor to ensure that these two activities occur on the last two meetings of your club, regardless of what Pacing Guide Day you are on in the 
last week. 


Appendix C 


Club Session Log Content 


Date 2/23/2/016 
Substituting? No 
Was Math Club Held Today? Yes 


Status Completed 
Club 401A 


Start Time 2/23/2016 11:01AM 
End Time 2/23/2016 11:26AM 


Lesson Number 43 


Present Partial Attendance Absent 
Student A O O O 
Student B O O O 
Student C 2 O O 
Student D O O 1 


Start-Up Activity? Yes Game Day? No 


Main Day Activity? Yes Assessment Day? 


Start-Up Activity Duration 


(Minutes) : 
First Start-Up Activity Core Unit Concert 
Second Start-Up Activity Go Round 
Engagement 
@ O O O @ 
1 2 3 4 ) 


All students are 
disengaged 


More students 
are disengaged 
than engaged 


Students are split 
equally 


More students 
are engaged than 
disengaged 


All students are 
engaged 
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Main Activity Duration 
(Minutes) 


Main Activity Comparison Bingo 


Engagement 
O O O O O 
1 2 3 4 5 
M . M tudent 
All students are a nue Students are split ieee All students are 
dea q are disengaged equall are engaged than shcndel 
i . 
ie than engaged cae disengaged ae 


Please indicate whether any of these problems were encountered today 


O) Transitioning children to clubs 


O Transition children from clubs 


QO) Difficult to use start-up material 


QO Difficult to use main activity material 


CO Low levels of student engagement 


OC) Not enough material for start-up 


OC Not enough material for main activity 


O Other 


Other Problems Description: 


Challenging Behavior 
oO O O O O 
1 2 3 4 5 
Intermittent/minor — Frequent/minor Somewhat ‘ 
; : : : Ongoing and 
No instances behavior chal- OR intermit- frequent and Paes 
lenges tent/intense intense 


Definitions for select terms and phrases from the Facilitator Log are provided below in the order 
in which they occur on the log. 


Basic Information 


e Date for which you are submitting a log: The day the club was held. 


e Substitute facilitator: Facilitator is facilitating a club they are not normally assigned to 
because the assigned facilitator is not available. 


e Location: The name of the school at which the club was held. 
e Group: The name associated with a specific club. 


e Math club held: Mark “yes” if a math club occurred. If not, select from one of the follow- 
ing: 


o Scheduled School Closing: There was a planned school closing due to the observance 
of national, local, or religious holidays, scheduled school breaks or teacher profes- 
sional development days. Neither children nor staff are in attendance on these days. 


o Unscheduled School Closing: The building is closed as a result of inclement weather, 
facilities malfunction or repair, or any other unforeseen reason. 


o Space Unavailable: No location was available to run the math club and therefore the 
facilitator was unable to conduct the club. 


o Change in School Schedule: A change in the student or teacher schedule prevented a 
math club from occurring (for example, if all kindergarten students are away on an 
all-day field trip). 


o Insufficient Number of Students Attended: The number of students in attendance was 
below two and only one adult was in attendance or no students were in attendance 
and therefore a math club could not be conducted. Do not select this option if no stu- 
dents were in attendance because of a change in the school schedule (see above). 


e High 5s Club Lesson #: The lesson number (indicated on the High 5s Pacing Guide) that 
was implemented that day. Please indicate the lesson you implemented that day even if you 
made a mistake and implemented a lesson that was scheduled to occur on a different day. 

e Start time: The time when instruction began. 


e End time: The time when students were dismissed. 


e Present: Student was present for the entirety of the club (from the start time to the end time). 
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Attendance 


o Partial Attendance: Student was present for at least 10 minutes, but was 5 or more 
minutes late, or left the club 5 or more minutes early. 


o Absent: Student was not present for at least 10 minutes of the club. 


Game Day: A Game Day was held during the club. Please indicate if you held a Game Day 
even if you made a mistake and according to the Pacing Guide a Game Day was not sched- 
uled to occur on that day. 


High 5s Club routines: The set of activities specified on the High 5s Pacing Guide, includ- 
ing Start-up and Main Activities. 


o Start-up: The set of activities, specified on the Pacing Guide, to be played at the start 
of a High 5s Club. Each lesson will include two Start-up Activities. Please indicate 
that Start-Up activities you implemented, even if you accidently implemented activi- 
ties that, according to the Pacing Guide, were not schedule to occur that day. 


o Main Activity: The activity specified on the Pacing Guide to be played after the 
Start-up activities. A lesson will include one Main Activity. Please indicate that Main 
Activity you implemented, even if you accidently implemented activities that, accord- 
ing to the Pacing Guide, were not schedule to occur that day. 
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Section A — Feedback on Start-Up Activities 


Start-up Activities played today: The Start-up Activities played in a particular club on a 
particular day, regardless of what should have been played according to the lesson in the Pac- 
ing Guide. 


Minutes spent on Start-up Activities: The total amount of time, in minutes, spent on both 
Start-up Activities. 


Student engagement in Start-up Activities: The overall level of student engagement during 
both Start-up activities, on a scale of 1 to 5, based on whether the children are engaged, re- 
sponsive, playing with materials, coming up with their own extensions to the activities, smil- 
ing and laughing, and want to continue beyond the time allotted. A score of | indicates that 
all students are disengaged while a score of 5 indicates that all students are engaged. 
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Section B — Feedback on Main Activity 


Main Activity played today: The Main Activity played in a particular club on a particular 
day, regardless of what should have been played according to the lesson in the Pacing Guide. 


Minutes spent on Main Activity: The total amount of time, in minutes, spent on the Main 
Activity. 


Student engagement in Main Activity: The overall level of student engagement during the 
Main Activity, on a scale of 1 to 5, based on whether the children are engaged, responsive, 
playing with materials, coming up with their own extensions to the activities, smiling and 
laughing, and want to continue beyond the time allotted. A score of 1 indicates that all stu- 
dents are disengaged while a score of 5 indicates that all students are engaged. 


78 


Section C — Game Day Activities 


e Student engagement in Game Day Activities: The overall level of student engagement 
during Game Day Activities, on a scale of 1 to 5, based on whether the children are engaged, 
responsive, playing with materials, coming up with their own extensions to the activities, 
smiling and laughing, and want to continue beyond the time allotted. A score of 1 indicates 
that all students are disengaged while a score of 5 indicates that all students are engaged. 


79 


Section D — Assessment Day 


e Assessment Day: Supervisor will collect formative assessment data on children’s counting 
and geometry knowledge three times a year. This information is not used for the research but 
will be used to help improve facilitators’ implementation by helping them to target the math 
activities to children’s level more effectively. 


o Assessment activity conducted: The assessment conducted as specified by the _. 


e Trajectory: A highly probable progression of learning through which students develop over 
time. Our focus on learning trajectories, rather than on content standards, allows us to align 
the lessons and facilitator interactions with students’ natural tendencies as they develop 
mathematically. Learning trajectories are based on the observed mathematical behavior of 
students, and each trajectory is focused on a single concept area. In the High 5s program, we 
have developed activities to support students in their learning in five concept areas; counting, 
composition of number, early addition and subtraction, constructing shapes, and attributes of 
shapes. 


o Trajectory level: On average, what a facilitator, supervisor, or other observer saw a 
child do in a particular activity during an assessment. See Section 3 of the manual for 
more information on trajectory levels. 
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Section E — Overall Feedback 


Problems: Any issues that hinder facilitation of a math club, force a facilitator to spent too 
much or too little time with a club, detract from the planned activities, reduce the quality of 
instruction, force a facilitator to veer off-script, lessen fidelity to the facilitation model, create 
distractions, reduce student engagement, etc. 


O 


Problems transitioning children to clubs: The facilitator or school staff had issues 
moving the children from their original locations to the High 5s club. 


Problems transitioning children from clubs back to their regular activities: The 
facilitator or school staff had issues transitioning the children from the High 5s club 
back to their regular school activities. 


Materials for the Start-up Activities were difficult to use: The materials intended 
for the Start-up Activity were designed in a way that made it difficult to facilitate the 
activity. 


Materials for the Main Activity were difficult to use: The materials intended for 
the Main Activity were designed in a way that made it difficult to facilitate the activi- 


ty. 


Low levels of student engagement: Student shows a low level of interest, involve- 
ment, and excitement around the activities. 


Not enough material written to fill the time for the Start-up Activities: The com- 
bined length of the Start-up Activities was too short to allow the facilitator to fill the - 
10-14 minutes allotted. 


Not enough material written to fill the time for the Main Activity: The combined 
length of the Main Activity was too short to allow the facilitator to fill the 20-30 
minutes allotted. 


Challenging behavior: Any student behavior that hinders facilitation of a math club by 
creating distractions for other students, reducing the quality of instruction, taking a signifi- 
cant portion of the facilitator’s attention, creating a hostile environment, or causing any other 
problems. A score of | indicates that there were no instances of behavior challenges while a 
score of 5 indicates that there were ongoing an intense instances of behavior challenges 
throughout the club. 
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About MDRC 


MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 


Founded in 1974 and located in New York; Oakland, California; Washington, DC; and Los 
Angeles, MDRC is best known for mounting rigorous, large-scale, real-world tests of new and 
existing policies and programs. Its projects are a mix of demonstrations (field tests of promising 
new program approaches) and evaluations of ongoing government and community initiatives. 
MDRC’s staff members bring an unusual combination of research and organizational experi- 
ence to their work, providing expertise on the latest in qualitative and quantitative methods and 
on program design, development, implementation, and management. MDRC seeks to learn not 
just whether a program is effective but also how and why the program’s effects occur. In addi- 
tion, it tries to place each project’s findings in the broader context of related research — in order 
to build knowledge about what works across the social and education policy fields. MDRC’s 
findings, lessons, and best practices are shared with a broad audience in the policy and practi- 
tioner community as well as with the general public and the media. 


Over the years, MDRC has brought its unique approach to an ever-growing range of policy are- 
as and target populations. Once known primarily for evaluations of state welfare-to-work pro- 
grams, today MDRC is also studying public school reforms, employment programs for ex- 
prisoners, and programs to help low-income students succeed in college. MDRC’s projects are 
organized into five areas: 

e Promoting Family Well-Being and Children’s Development 

e Improving Public Education 

e Raising Academic Achievement and Persistence in College 

e Supporting Low-Wage Workers and Communities 


e Overcoming Barriers to Employment 


Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 


